Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250905 07:49:58.869177 1738 test_util.cc:276] Using random seed: 362977423
2025-09-05T07:49:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T07:49:58Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task7ZTeCZ/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task7ZTeCZ/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/security.properties
2025-09-05 07:50:05: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task7ZTeCZ/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task7ZTeCZ/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250905 07:50:15.081430 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--webserver_interface=127.1.178.190
--webserver_port=0
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:15.394294 1981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:15.394697 1981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:15.394834 1981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:15.403010 1981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:15.403124 1981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:15.403214 1981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:15.403282 1981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:15.413092 1981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.178.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:15.414781 1981 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:15.416541 1981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:15.424998 1987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:15.425176 1989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:15.424970 1986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:15.425631 1981 server_base.cc:1047] running on GCE node
I20250905 07:50:15.426481 1981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:15.427892 1981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:15.429159 1981 hybrid_clock.cc:648] HybridClock initialized: now 1757058615429110 us; error 46 us; skew 500 ppm
I20250905 07:50:15.429510 1981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:15.431615 1981 webserver.cc:480] Webserver started at http://127.1.178.190:34789/ using document root <none> and password file <none>
I20250905 07:50:15.432402 1981 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:15.432516 1981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:15.432874 1981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:50:15.435274 1981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/instance:
uuid: "9dbf15725a7342da830f57c0535eb72d"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:15.436110 1981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal/instance:
uuid: "9dbf15725a7342da830f57c0535eb72d"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:15.441331 1981 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250905 07:50:15.444931 1995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:15.446266 1981 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:50:15.446439 1981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
uuid: "9dbf15725a7342da830f57c0535eb72d"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:15.446635 1981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:15.494297 1981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:15.495239 1981 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:15.495522 1981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:15.515892 1981 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.190:35357
I20250905 07:50:15.515947 2046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.190:35357 every 8 connection(s)
I20250905 07:50:15.517441 1981 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
I20250905 07:50:15.520458 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 1981
I20250905 07:50:15.520782 1738 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal/instance
I20250905 07:50:15.522490 2047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:15.524284 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--webserver_interface=127.1.178.189
--webserver_port=0
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 07:50:15.530686 2047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
I20250905 07:50:15.550560 2047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
W20250905 07:50:15.551591 1996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
W20250905 07:50:15.553241 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.189:43391: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:15.602667 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } attempt: 1
W20250905 07:50:15.604662 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.189:43391: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:15.673028 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } attempt: 2
W20250905 07:50:15.674840 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.189:43391: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:15.766196 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } attempt: 3
W20250905 07:50:15.768057 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.189:43391: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
W20250905 07:50:15.858111 2048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:15.858472 2048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:15.858600 2048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:15.866909 2048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:15.867132 2048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:15.867260 2048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:15.867331 2048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:15.877548 2048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.178.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:15.879248 2048 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:15.881124 2048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:15.889583 2056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:15.889533 2057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:15.890820 2059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:15.890941 2048 server_base.cc:1047] running on GCE node
I20250905 07:50:15.891769 2048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:15.893288 2048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:15.894511 2048 hybrid_clock.cc:648] HybridClock initialized: now 1757058615894483 us; error 71 us; skew 500 ppm
I20250905 07:50:15.894884 2048 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:15.897208 2048 webserver.cc:480] Webserver started at http://127.1.178.189:37293/ using document root <none> and password file <none>
I20250905 07:50:15.898011 2048 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:15.898133 2048 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:15.898515 2048 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:50:15.901006 2048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/instance:
uuid: "65033f3e4b44455389e29a5e1c9f82cd"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:15.901818 2048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal/instance:
uuid: "65033f3e4b44455389e29a5e1c9f82cd"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:15.907179 2048 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20250905 07:50:15.910593 2065 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:15.911283 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } attempt: 4
I20250905 07:50:15.911999 2048 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250905 07:50:15.912180 2048 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
uuid: "65033f3e4b44455389e29a5e1c9f82cd"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:15.912398 2048 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250905 07:50:15.913134 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.189:43391: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:15.943512 2048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:15.944475 2048 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:15.944754 2048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:15.965059 2048 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.189:43391
I20250905 07:50:15.965080 2116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.189:43391 every 8 connection(s)
I20250905 07:50:15.966607 2048 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
I20250905 07:50:15.970707 2117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:15.973511 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 2048
I20250905 07:50:15.973691 1738 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal/instance
I20250905 07:50:15.974761 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.188:39743
--webserver_interface=127.1.178.188
--webserver_port=0
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 07:50:15.976660 2117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
I20250905 07:50:15.989502 2117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
I20250905 07:50:15.999159 2117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } has no permanent_uuid. Determining permanent_uuid...
W20250905 07:50:16.000249 2068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:16.001751 2117 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:16.051129 2117 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 1
W20250905 07:50:16.052901 2117 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:16.121109 2117 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 2
W20250905 07:50:16.122866 2117 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:16.212389 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } attempt: 5
I20250905 07:50:16.214105 2117 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 3
W20250905 07:50:16.215880 2117 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:16.219260 2047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } has no permanent_uuid. Determining permanent_uuid...
W20250905 07:50:16.221093 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:16.272302 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 1
W20250905 07:50:16.274233 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:16.304101 2118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:16.304455 2118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:16.304580 2118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:16.313832 2118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:16.313958 2118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:16.314040 2118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:16.314109 2118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:16.325788 2118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.188:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.178.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:16.327487 2118 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:16.329339 2118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:16.337173 2126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:16.337227 2127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:16.337682 2129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:16.338228 2118 server_base.cc:1047] running on GCE node
I20250905 07:50:16.338979 2118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:16.340365 2118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:16.341593 2118 hybrid_clock.cc:648] HybridClock initialized: now 1757058616341552 us; error 31 us; skew 500 ppm
I20250905 07:50:16.341924 2118 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:16.342466 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 2
W20250905 07:50:16.344388 2047 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:16.344457 2118 webserver.cc:480] Webserver started at http://127.1.178.188:42041/ using document root <none> and password file <none>
I20250905 07:50:16.345166 2118 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:16.345252 2118 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:16.345538 2118 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:50:16.347812 2118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/instance:
uuid: "81e2a8aedb144e61891de4dcd390ff35"
format_stamp: "Formatted at 2025-09-05 07:50:16 on dist-test-slave-tgv6"
I20250905 07:50:16.348603 2118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal/instance:
uuid: "81e2a8aedb144e61891de4dcd390ff35"
format_stamp: "Formatted at 2025-09-05 07:50:16 on dist-test-slave-tgv6"
I20250905 07:50:16.353595 2118 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 07:50:16.356693 2135 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:16.357937 2118 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:50:16.358098 2118 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
uuid: "81e2a8aedb144e61891de4dcd390ff35"
format_stamp: "Formatted at 2025-09-05 07:50:16 on dist-test-slave-tgv6"
I20250905 07:50:16.358274 2118 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:16.359139 2117 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 4
W20250905 07:50:16.360960 2117 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:16.375478 2118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:16.376389 2118 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:16.376657 2118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:16.400048 2118 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.188:39743
I20250905 07:50:16.400075 2186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.188:39743 every 8 connection(s)
I20250905 07:50:16.401695 2118 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
I20250905 07:50:16.401988 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 2118
I20250905 07:50:16.402199 1738 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal/instance
I20250905 07:50:16.407294 2187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:16.417021 2187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
I20250905 07:50:16.434135 2187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
I20250905 07:50:16.442351 2187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } has no permanent_uuid. Determining permanent_uuid...
I20250905 07:50:16.450619 2047 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 3
I20250905 07:50:16.453116 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Bootstrap starting.
I20250905 07:50:16.457535 2187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:16.459070 2187 log.cc:826] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:16.462951 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: No bootstrap required, opened a new log
I20250905 07:50:16.468578 2187 raft_consensus.cc:357] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.468909 2187 raft_consensus.cc:383] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:16.469025 2187 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Initialized, Role: FOLLOWER
I20250905 07:50:16.469727 2047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap starting.
I20250905 07:50:16.469762 2187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.470945 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.471215 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:16.472434 2187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:16.474279 2047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:16.475847 2047 log.cc:826] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:16.479215 2047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: No bootstrap required, opened a new log
I20250905 07:50:16.487161 2047 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.487469 2047 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:16.487567 2047 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Initialized, Role: FOLLOWER
I20250905 07:50:16.488263 2047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.489306 2211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.489674 2211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:16.490615 2047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: configured and running, proceeding with master startup.
W20250905 07:50:16.491688 2208 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:50:16.491859 2208 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:50:16.485687 2118 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: OK, instance_id { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" instance_seqno: 1757058615510348 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250905 07:50:16.493614 2118 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:50:16.498347 2001 tablet.cc:2378] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:50:16.512967 2225 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 07:50:16.513235 2225 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 07:50:16.508976 1981 master_runner.cc:186] Error getting master registration for 127.1.178.189:43391: OK, instance_id { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" instance_seqno: 1757058615959588 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250905 07:50:16.515832 1981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:16.583071 2193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:16.583396 2193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.585201 2193 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
I20250905 07:50:16.586521 2022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d" is_pre_election: true
I20250905 07:50:16.587132 2022 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 0.
I20250905 07:50:16.588480 2136 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d; no voters:
I20250905 07:50:16.588948 2193 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:16.589112 2193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:16.589231 2193 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:16.592949 2193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.598075 2022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d"
I20250905 07:50:16.598305 2022 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:16.601425 2022 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 1.
I20250905 07:50:16.602056 2136 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d; no voters:
I20250905 07:50:16.602871 2193 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 1 election: Requested vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
I20250905 07:50:16.603549 2193 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:16.604354 2193 raft_consensus.cc:695] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 1 LEADER]: Becoming Leader. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Running, Role: LEADER
I20250905 07:50:16.606607 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
W20250905 07:50:16.628676 2141 tablet.cc:2378] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:50:16.630410 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 1 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.630746 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:16.640575 2230 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:16.642891 2230 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:16.660146 2117 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } attempt: 5
I20250905 07:50:16.666214 2022 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 1 FOLLOWER]: Refusing update from remote peer 81e2a8aedb144e61891de4dcd390ff35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:50:16.667315 2193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:16.680038 2117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap starting.
I20250905 07:50:16.702993 2117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:16.712436 2117 log.cc:826] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:16.720312 2211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 1 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.720651 2211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:16.734035 2117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: No bootstrap required, opened a new log
I20250905 07:50:16.741876 2211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.742120 2211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:16.745630 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.745851 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:16.748147 2230 catalog_manager.cc:1349] Generated new cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:16.747758 2117 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.748246 2230 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:16.748425 2117 raft_consensus.cc:383] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:16.748620 2117 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Initialized, Role: FOLLOWER
I20250905 07:50:16.749895 2117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.780613 2238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.786459 2237 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:16.792053 2238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:16.787240 2117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:16.787292 2091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:16.787909 2092 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:16.792220 2237 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:16.798000 2090 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:16.799479 2021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d" is_pre_election: true
I20250905 07:50:16.800406 2237 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:16.801523 2090 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 1 FOLLOWER]: Refusing update from remote peer 81e2a8aedb144e61891de4dcd390ff35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 07:50:16.803064 2228 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:16.803341 2162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
I20250905 07:50:16.804484 2068 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:16.804824 2237 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:50:16.836475 2238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.836757 2238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:16.836810 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.837033 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:16.836464 2237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 1 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:16.838156 2237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:16.842129 2230 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 07:50:16.844347 2230 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:16.864512 2230 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Generated new TSK 0
I20250905 07:50:16.865999 2230 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:16.867588 2230 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:16.868885 2230 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:16.880189 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.178.129:0
--local_ip_for_outbound_sockets=127.1.178.129
--webserver_interface=127.1.178.129
--webserver_port=0
--tserver_master_addrs=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250905 07:50:16.907472 2256 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:16.907689 2256 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: loading cluster ID for follower catalog manager: success
I20250905 07:50:16.911903 2256 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: acquiring CA information for follower catalog manager: success
I20250905 07:50:16.922616 2256 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250905 07:50:17.346988 2257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:17.347328 2257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:17.347470 2257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:17.355947 2257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:17.356197 2257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.178.129
I20250905 07:50:17.366515 2257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.178.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.178.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.178.129
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:17.368505 2257 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:17.370519 2257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:17.380754 2264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:17.381008 2266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:17.380862 2263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:17.381681 2257 server_base.cc:1047] running on GCE node
I20250905 07:50:17.382231 2257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:17.383121 2257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:17.384313 2257 hybrid_clock.cc:648] HybridClock initialized: now 1757058617384258 us; error 44 us; skew 500 ppm
I20250905 07:50:17.384617 2257 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:17.386658 2257 webserver.cc:480] Webserver started at http://127.1.178.129:41153/ using document root <none> and password file <none>
I20250905 07:50:17.387398 2257 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:17.387507 2257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:17.387848 2257 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:50:17.390249 2257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data/instance:
uuid: "b902ef48db944a02813b8543b2f91a7e"
format_stamp: "Formatted at 2025-09-05 07:50:17 on dist-test-slave-tgv6"
I20250905 07:50:17.391012 2257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/instance:
uuid: "b902ef48db944a02813b8543b2f91a7e"
format_stamp: "Formatted at 2025-09-05 07:50:17 on dist-test-slave-tgv6"
I20250905 07:50:17.396519 2257 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250905 07:50:17.399752 2272 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:17.401111 2257 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:50:17.401260 2257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal
uuid: "b902ef48db944a02813b8543b2f91a7e"
format_stamp: "Formatted at 2025-09-05 07:50:17 on dist-test-slave-tgv6"
I20250905 07:50:17.401460 2257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:17.420532 2257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:17.421319 2257 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:17.421618 2257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:17.422556 2257 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:50:17.424216 2257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:50:17.424307 2257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:17.424432 2257 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:50:17.424512 2257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:17.456601 2257 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.129:45773
I20250905 07:50:17.456665 2384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.129:45773 every 8 connection(s)
I20250905 07:50:17.458034 2257 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/data/info.pb
I20250905 07:50:17.462821 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 2257
I20250905 07:50:17.463028 1738 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/instance
I20250905 07:50:17.469477 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.178.130:0
--local_ip_for_outbound_sockets=127.1.178.130
--webserver_interface=127.1.178.130
--webserver_port=0
--tserver_master_addrs=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250905 07:50:17.491236 2385 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:17.491679 2385 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:17.492849 2387 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:17.493000 2387 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:17.493044 2385 heartbeater.cc:507] Master 127.1.178.188:39743 requested a full tablet report, sending...
I20250905 07:50:17.493571 2387 heartbeater.cc:507] Master 127.1.178.189:43391 requested a full tablet report, sending...
I20250905 07:50:17.495715 2152 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:17.497493 2081 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:17.498049 2152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.178.129:47759
I20250905 07:50:17.499313 2386 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:17.499491 2386 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:17.500146 2386 heartbeater.cc:507] Master 127.1.178.190:35357 requested a full tablet report, sending...
I20250905 07:50:17.501966 2012 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:17.515687 2225 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:17.515792 2225 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: loading cluster ID for follower catalog manager: success
I20250905 07:50:17.520473 2225 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: acquiring CA information for follower catalog manager: success
I20250905 07:50:17.526526 2225 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:17.768788 2396 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:17.769017 2396 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:17.769783 2396 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:17.770388 2162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
I20250905 07:50:17.770377 2021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d" is_pre_election: true
I20250905 07:50:17.771040 2066 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:17.771423 2396 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 07:50:17.793658 2397 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:17.793926 2397 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:17.795262 2397 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:17.795853 2090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:17.796017 2162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
I20250905 07:50:17.796156 2090 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 1.
I20250905 07:50:17.796650 1996 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters:
I20250905 07:50:17.797067 2397 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 07:50:17.797191 2397 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:17.797299 2397 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 1 FOLLOWER]: Advancing to term 2
I20250905 07:50:17.799374 2397 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:17.800055 2397 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 2 election: Requested vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:17.800639 2090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:17.800686 2162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35"
I20250905 07:50:17.800870 2090 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 1 FOLLOWER]: Advancing to term 2
I20250905 07:50:17.802933 2090 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 2.
I20250905 07:50:17.803386 1996 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:17.803716 2397 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 2 FOLLOWER]: Leader election won for term 2
I20250905 07:50:17.804188 2397 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 2 LEADER]: Becoming Leader. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Running, Role: LEADER
I20250905 07:50:17.804829 2397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:17.807183 2398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 2 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:17.807428 2021 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 2 LEADER]: Rejecting Update request from peer 81e2a8aedb144e61891de4dcd390ff35 for earlier term 1. Current term is 2. Ops: []
I20250905 07:50:17.807520 2398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:17.808198 2399 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 07:50:17.808463 2400 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:17.811589 2400 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:17.812096 2399 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Peer 9dbf15725a7342da830f57c0535eb72d log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250905 07:50:17.812539 2401 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 07:50:17.812669 2401 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Running, Role: LEADER
I20250905 07:50:17.812885 2400 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:17.812984 2400 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:17.813133 2401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:17.813791 2401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 1 FOLLOWER]: Advancing to term 2
I20250905 07:50:17.814237 2400 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:17.815063 2400 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Loaded TSK: 0
I20250905 07:50:17.816015 2400 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:17.816844 2400 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:17.817677 2400 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
W20250905 07:50:17.839835 2395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:17.840220 2395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:17.840371 2395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:17.848665 2395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:17.848912 2395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.178.130
I20250905 07:50:17.858498 2395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.178.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.178.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.178.130
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:17.860183 2395 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:17.861922 2395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:17.871623 2407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:17.871726 2406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:17.871742 2409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:17.873230 2395 server_base.cc:1047] running on GCE node
I20250905 07:50:17.873780 2395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:17.874708 2395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:17.875988 2395 hybrid_clock.cc:648] HybridClock initialized: now 1757058617875878 us; error 46 us; skew 500 ppm
I20250905 07:50:17.876299 2395 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:17.878568 2395 webserver.cc:480] Webserver started at http://127.1.178.130:45477/ using document root <none> and password file <none>
I20250905 07:50:17.879345 2395 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:17.879441 2395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:17.879810 2395 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:50:17.882316 2395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data/instance:
uuid: "0f0103de40bc4724a20d7dc4bff02114"
format_stamp: "Formatted at 2025-09-05 07:50:17 on dist-test-slave-tgv6"
I20250905 07:50:17.883097 2395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/instance:
uuid: "0f0103de40bc4724a20d7dc4bff02114"
format_stamp: "Formatted at 2025-09-05 07:50:17 on dist-test-slave-tgv6"
I20250905 07:50:17.888686 2395 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250905 07:50:17.892412 2415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:17.893965 2395 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:50:17.894109 2395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal
uuid: "0f0103de40bc4724a20d7dc4bff02114"
format_stamp: "Formatted at 2025-09-05 07:50:17 on dist-test-slave-tgv6"
I20250905 07:50:17.894282 2395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:17.917032 2395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:17.917902 2395 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:17.918175 2395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:17.919116 2395 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:50:17.920831 2395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:50:17.920920 2395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:17.921003 2395 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:50:17.921061 2395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:17.954553 2395 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.130:37221
I20250905 07:50:17.954646 2527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.130:37221 every 8 connection(s)
I20250905 07:50:17.956166 2395 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/data/info.pb
I20250905 07:50:17.965060 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 2395
I20250905 07:50:17.965258 1738 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/instance
I20250905 07:50:17.975500 2528 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:17.975853 2528 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:17.976212 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.178.131:0
--local_ip_for_outbound_sockets=127.1.178.131
--webserver_interface=127.1.178.131
--webserver_port=0
--tserver_master_addrs=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250905 07:50:17.976804 2528 heartbeater.cc:507] Master 127.1.178.188:39743 requested a full tablet report, sending...
I20250905 07:50:17.978343 2152 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:17.979597 2530 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:17.979750 2530 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:17.980274 2530 heartbeater.cc:507] Master 127.1.178.189:43391 requested a full tablet report, sending...
I20250905 07:50:17.980867 2529 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:17.980994 2529 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:17.981500 2529 heartbeater.cc:507] Master 127.1.178.190:35357 requested a full tablet report, sending...
I20250905 07:50:17.981612 2081 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:17.982343 2012 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:17.984587 2012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.178.130:53127
I20250905 07:50:18.001950 2385 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
I20250905 07:50:18.287606 2401 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:18.287842 2401 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:18.288750 2401 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
I20250905 07:50:18.289227 2021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d" is_pre_election: true
I20250905 07:50:18.289323 2090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:18.289575 2090 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 2.
I20250905 07:50:18.289973 2136 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:18.290232 2401 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 07:50:18.290349 2401 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:18.290441 2401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 2 FOLLOWER]: Advancing to term 3
I20250905 07:50:18.292469 2401 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:18.293502 2401 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 3 election: Requested vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
I20250905 07:50:18.293730 2021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d"
I20250905 07:50:18.293974 2090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:18.294250 2090 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 2 FOLLOWER]: Advancing to term 3
I20250905 07:50:18.296276 2090 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 3.
I20250905 07:50:18.296697 2136 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:18.296975 2401 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 07:50:18.297122 2401 raft_consensus.cc:695] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 LEADER]: Becoming Leader. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Running, Role: LEADER
I20250905 07:50:18.297405 2401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:18.298666 2399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 3 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:18.298898 2399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:18.299436 2540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:18.301321 2540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:18.302217 2540 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:18.302297 2540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:18.303587 2540 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:18.304499 2540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Loaded TSK: 0
W20250905 07:50:18.304801 2535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:18.305099 2535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:18.305238 2535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250905 07:50:18.305236 2540 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:18.305964 2540 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:18.306704 2540 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
W20250905 07:50:18.313952 2535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:18.314193 2535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.178.131
I20250905 07:50:18.324290 2535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.178.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.178.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.178.131
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:18.325994 2535 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:18.327788 2535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:18.338075 2542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:18.337998 2543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:18.338601 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:18.339148 2535 server_base.cc:1047] running on GCE node
I20250905 07:50:18.339745 2535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:18.341480 2535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:18.342674 2535 hybrid_clock.cc:648] HybridClock initialized: now 1757058618342632 us; error 42 us; skew 500 ppm
I20250905 07:50:18.342981 2535 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:18.345090 2535 webserver.cc:480] Webserver started at http://127.1.178.131:45133/ using document root <none> and password file <none>
I20250905 07:50:18.345827 2535 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:18.345921 2535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:18.346264 2535 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 07:50:18.348718 2535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data/instance:
uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
format_stamp: "Formatted at 2025-09-05 07:50:18 on dist-test-slave-tgv6"
I20250905 07:50:18.349505 2535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/instance:
uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
format_stamp: "Formatted at 2025-09-05 07:50:18 on dist-test-slave-tgv6"
I20250905 07:50:18.354728 2535 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 07:50:18.357945 2551 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:18.359258 2535 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:50:18.359419 2535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal
uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
format_stamp: "Formatted at 2025-09-05 07:50:18 on dist-test-slave-tgv6"
I20250905 07:50:18.359618 2535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:18.376268 2535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:18.377130 2535 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:18.377403 2535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:18.378310 2535 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 07:50:18.380012 2535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 07:50:18.380110 2535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:18.380232 2535 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 07:50:18.380306 2535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:18.390089 2162 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 LEADER]: Rejecting Update request from peer 9dbf15725a7342da830f57c0535eb72d for earlier term 2. Current term is 3. Ops: []
I20250905 07:50:18.390930 2576 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 07:50:18.391952 2576 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Peer 81e2a8aedb144e61891de4dcd390ff35 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250905 07:50:18.392266 2586 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 2 LEADER]: Stepping down as leader of term 2
I20250905 07:50:18.392385 2586 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Running, Role: LEADER
I20250905 07:50:18.392894 2586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:18.393469 2586 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 2 FOLLOWER]: Advancing to term 3
I20250905 07:50:18.417322 2535 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.131:45275
I20250905 07:50:18.417407 2665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.131:45275 every 8 connection(s)
I20250905 07:50:18.418871 2535 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/data/info.pb
I20250905 07:50:18.425238 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 2535
I20250905 07:50:18.425472 1738 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/instance
I20250905 07:50:18.438215 2666 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:18.438589 2666 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:18.439015 2668 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:18.439182 2668 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:18.439905 2666 heartbeater.cc:507] Master 127.1.178.188:39743 requested a full tablet report, sending...
I20250905 07:50:18.439972 2668 heartbeater.cc:507] Master 127.1.178.189:43391 requested a full tablet report, sending...
I20250905 07:50:18.440052 2667 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:18.440228 2667 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:18.440730 2667 heartbeater.cc:507] Master 127.1.178.190:35357 requested a full tablet report, sending...
I20250905 07:50:18.441810 2152 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:18.441913 2081 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:18.442561 2012 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:18.443230 2152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.178.131:45801
I20250905 07:50:18.454169 1738 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 07:50:18.485505 2152 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_cc7e423f936848e1b8abbb22c3440719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.485540 2151 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_45c1508d77dc4693960b671c9894a5b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.487512 2149 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_f040e3a613e6480bb336e73085713781"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.488062 2529 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:18.488634 2150 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.489256 2148 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_fbd6c4d89fd54fdd9765576cc3cf5cbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.489821 2147 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_146568e744294fdd8f082cdbd8451039"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.490208 2146 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_a908ec5ae1aa46e8b872ff681d610476"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.490590 2145 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_df9d719332a94674bf8d0b4f1f9d6a7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.490972 2144 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_9a0733038d2145b2a3ea38308d4b6fa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.491322 2143 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45672:
name: "default.Table_18a839c38dff41c892ed79bdaf6c8229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:50:18.498435 2152 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cc7e423f936848e1b8abbb22c3440719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:50:18.499775 1738 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu pid 2118
W20250905 07:50:18.500699 2151 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_45c1508d77dc4693960b671c9894a5b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:50:18.500998 2182 generic_service.cc:196] Checking for leaks (request via RPC)
I20250905 07:50:18.505782 2386 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:18.673866 1738 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu with pid 2118
W20250905 07:50:18.702883 2675 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.188:39743)
W20250905 07:50:18.708096 2385 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.1.178.188:39743 (error 108)
W20250905 07:50:18.711627 2679 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:50:18.704465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.178.188:39743 (local address 127.0.0.1:45674)
0905 07:50:18.705304 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0905 07:50:18.705328 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0905 07:50:18.705532 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:50:18.709985 (+ 4453us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":556}
I20250905 07:50:18.801652 2586 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:18.801867 2586 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:18.802690 2586 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:18.803133 2090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:18.803357 2090 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 3.
I20250905 07:50:18.803803 1996 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters:
I20250905 07:50:18.804234 2586 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 07:50:18.804366 2586 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:18.804484 2586 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 3 FOLLOWER]: Advancing to term 4
W20250905 07:50:18.805392 1998 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:18.807298 2586 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:18.807844 2586 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 4 election: Requested vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:18.808501 2090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:18.808703 2090 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 3 FOLLOWER]: Advancing to term 4
I20250905 07:50:18.811528 2090 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 4.
W20250905 07:50:18.811740 1998 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:18.813237 1996 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:18.813767 2586 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 FOLLOWER]: Leader election won for term 4
I20250905 07:50:18.813938 2586 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 LEADER]: Becoming Leader. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Running, Role: LEADER
I20250905 07:50:18.814250 2586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:18.815747 2576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 4 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:18.815971 2576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:18.816635 2716 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:18.818781 2716 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:18.819787 2716 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:18.819877 2716 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:18.821303 2716 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:18.822261 2716 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Loaded TSK: 0
I20250905 07:50:18.822808 2716 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:18.823711 2716 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:18.824647 2716 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:18.832758 2011 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_9a0733038d2145b2a3ea38308d4b6fa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.832894 2010 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_df9d719332a94674bf8d0b4f1f9d6a7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.834024 2012 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_18a839c38dff41c892ed79bdaf6c8229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.833014 2006 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.832777 2008 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_45c1508d77dc4693960b671c9894a5b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.834805 2009 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_a908ec5ae1aa46e8b872ff681d610476"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.837306 2004 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_fbd6c4d89fd54fdd9765576cc3cf5cbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.836269 2007 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_cc7e423f936848e1b8abbb22c3440719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.836885 2005 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_f040e3a613e6480bb336e73085713781"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:18.839232 2003 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_146568e744294fdd8f082cdbd8451039"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 07:50:18.844991 2010 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_df9d719332a94674bf8d0b4f1f9d6a7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.849851 2011 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9a0733038d2145b2a3ea38308d4b6fa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.850992 2003 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_146568e744294fdd8f082cdbd8451039 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.852035 2009 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a908ec5ae1aa46e8b872ff681d610476 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.853013 2004 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.853917 2005 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f040e3a613e6480bb336e73085713781 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.854872 2007 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cc7e423f936848e1b8abbb22c3440719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.855859 2008 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_45c1508d77dc4693960b671c9894a5b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.856797 2006 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 07:50:18.857734 2012 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_18a839c38dff41c892ed79bdaf6c8229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 07:50:18.945470 2666 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
W20250905 07:50:18.946832 2666 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:18.977070 2010 catalog_manager.cc:2149] created HMS catalog entry for table [id=07dcc92b52df4317b60c8f9acf93d938]
W20250905 07:50:18.982412 1998 consensus_peers.cc:489] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d -> Peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Couldn't send request to peer 81e2a8aedb144e61891de4dcd390ff35. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:18.983700 2090 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 4 FOLLOWER]: Refusing update from remote peer 9dbf15725a7342da830f57c0535eb72d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250905 07:50:18.983739 2528 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
I20250905 07:50:18.985055 2586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20250905 07:50:18.987344 2528 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:18.999269 2720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 4 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:18.999558 2720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:19.001909 2718 mvcc.cc:204] Tried to move back new op lower bound from 7196912103339106304 to 7196912098533634048. Current Snapshot: MvccSnapshot[applied={T|T < 7196912103339106304}]
I20250905 07:50:19.024681 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:19.025027 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:19.032223 2011 catalog_manager.cc:2149] created HMS catalog entry for table [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:19.035483 2010 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "default.Table_df9d719332a94674bf8d0b4f1f9d6a7f" } new_table_name: "default.Table_30a6668a12364eedb06bd2b9bfbf4b31" modify_external_catalogs: true
I20250905 07:50:19.044687 2724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:19.045076 2724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:19.045506 2720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:19.045722 2720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:19.084273 2003 catalog_manager.cc:2149] created HMS catalog entry for table [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:19.085548 2011 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "default.Table_9a0733038d2145b2a3ea38308d4b6fa8" } new_table_name: "default.Table_96eada578e0d465cb55ab0848c68acaa" modify_external_catalogs: true
I20250905 07:50:19.089877 2601 tablet_service.cc:1468] Processing CreateTablet for tablet af9984876006444aa460afd1604eb359 (DEFAULT_TABLE table=default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.091958 2601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9984876006444aa460afd1604eb359. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.101466 2463 tablet_service.cc:1468] Processing CreateTablet for tablet af9984876006444aa460afd1604eb359 (DEFAULT_TABLE table=default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.103566 2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9984876006444aa460afd1604eb359. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.121598 2320 tablet_service.cc:1468] Processing CreateTablet for tablet af9984876006444aa460afd1604eb359 (DEFAULT_TABLE table=default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.123427 2320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9984876006444aa460afd1604eb359. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.125530 2462 tablet_service.cc:1468] Processing CreateTablet for tablet 0c1289e4420649eb99ac6102e721db0c (DEFAULT_TABLE table=default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.126129 2462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1289e4420649eb99ac6102e721db0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.127543 2003 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "default.Table_146568e744294fdd8f082cdbd8451039" } new_table_name: "default.Table_bcc153425f724fa0aee1ba610df560ce" modify_external_catalogs: true
I20250905 07:50:19.129772 2600 tablet_service.cc:1468] Processing CreateTablet for tablet 0c1289e4420649eb99ac6102e721db0c (DEFAULT_TABLE table=default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.130035 2319 tablet_service.cc:1468] Processing CreateTablet for tablet 0c1289e4420649eb99ac6102e721db0c (DEFAULT_TABLE table=default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.130198 2600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1289e4420649eb99ac6102e721db0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.130800 2319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1289e4420649eb99ac6102e721db0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.147980 2318 tablet_service.cc:1468] Processing CreateTablet for tablet f6d24d267ac142fca1cec34552e395fd (DEFAULT_TABLE table=default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.148631 2318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d24d267ac142fca1cec34552e395fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.152912 2734 tablet_bootstrap.cc:492] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.153813 2733 tablet_bootstrap.cc:492] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.157017 2461 tablet_service.cc:1468] Processing CreateTablet for tablet f6d24d267ac142fca1cec34552e395fd (DEFAULT_TABLE table=default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.157646 2461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d24d267ac142fca1cec34552e395fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.163193 2735 tablet_bootstrap.cc:492] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.157317 2601 tablet_service.cc:1468] Processing CreateTablet for tablet f6d24d267ac142fca1cec34552e395fd (DEFAULT_TABLE table=default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.163828 2601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d24d267ac142fca1cec34552e395fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.167315 2735 tablet_bootstrap.cc:654] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.168840 2735 log.cc:826] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:19.175675 2734 tablet_bootstrap.cc:654] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.176568 2733 tablet_bootstrap.cc:654] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.177629 2734 log.cc:826] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:19.178071 2733 log.cc:826] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:19.188656 2009 catalog_manager.cc:2149] created HMS catalog entry for table [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:19.190513 2734 tablet_bootstrap.cc:492] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.190831 2734 ts_tablet_manager.cc:1397] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.000s
I20250905 07:50:19.199081 2733 tablet_bootstrap.cc:492] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.199415 2733 ts_tablet_manager.cc:1397] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.012s
I20250905 07:50:19.202256 2009 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c1289e4420649eb99ac6102e721db0c from {username='slave'} at 127.0.0.1:35886
I20250905 07:50:19.198171 2734 raft_consensus.cc:357] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.204931 2734 raft_consensus.cc:383] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.205071 2734 raft_consensus.cc:738] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.206961 2733 raft_consensus.cc:357] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.207671 2733 raft_consensus.cc:383] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.207850 2733 raft_consensus.cc:738] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.208547 2735 tablet_bootstrap.cc:492] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.208870 2735 ts_tablet_manager.cc:1397] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.056s user 0.018s sys 0.008s
I20250905 07:50:19.208781 2733 consensus_queue.cc:260] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.206032 2734 consensus_queue.cc:260] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.217391 2733 ts_tablet_manager.cc:1428] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.018s user 0.015s sys 0.002s
I20250905 07:50:19.218075 2529 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:19.220364 2733 tablet_bootstrap.cc:492] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.221642 2009 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet 0c1289e4420649eb99ac6102e721db0c
I20250905 07:50:19.223824 2733 tablet_bootstrap.cc:654] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.215858 2735 raft_consensus.cc:357] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.225026 2735 raft_consensus.cc:383] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.225209 2735 raft_consensus.cc:738] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.226635 2735 consensus_queue.cc:260] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.229068 2386 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:19.229980 2735 ts_tablet_manager.cc:1428] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.021s user 0.011s sys 0.000s
I20250905 07:50:19.230226 2461 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c1289e4420649eb99ac6102e721db0c with delete_type TABLET_DATA_DELETED (replaced by tablet 1a57e0ff6b6a422aa0e69678b7bd6762) from {username='slave'} at 127.0.0.1:52248
I20250905 07:50:19.230937 2319 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c1289e4420649eb99ac6102e721db0c with delete_type TABLET_DATA_DELETED (replaced by tablet 1a57e0ff6b6a422aa0e69678b7bd6762) from {username='slave'} at 127.0.0.1:52948
I20250905 07:50:19.231014 2667 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:19.234499 2735 tablet_bootstrap.cc:492] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.235155 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.188:39743
--webserver_interface=127.1.178.188
--webserver_port=42041
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 07:50:19.236758 2735 tablet_bootstrap.cc:654] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.237011 2009 catalog_manager.cc:6652] ReplaceTablet: tablet 0c1289e4420649eb99ac6102e721db0c deleted and replaced by tablet 1a57e0ff6b6a422aa0e69678b7bd6762
I20250905 07:50:19.238760 2733 tablet_bootstrap.cc:492] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.238952 2733 ts_tablet_manager.cc:1397] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250905 07:50:19.240125 2733 raft_consensus.cc:357] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.240388 2733 raft_consensus.cc:383] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.240483 2733 raft_consensus.cc:738] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.240782 2733 consensus_queue.cc:260] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.242570 2733 ts_tablet_manager.cc:1428] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 07:50:19.242908 2733 tablet_bootstrap.cc:492] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.244254 2734 ts_tablet_manager.cc:1428] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.053s user 0.019s sys 0.004s
I20250905 07:50:19.245875 2733 tablet_bootstrap.cc:654] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.247504 2734 tablet_bootstrap.cc:492] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.247578 2463 tablet_service.cc:1468] Processing CreateTablet for tablet 8b52a829672943da8121d121052697ab (DEFAULT_TABLE table=default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.248103 2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b52a829672943da8121d121052697ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.248251 2601 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c1289e4420649eb99ac6102e721db0c with delete_type TABLET_DATA_DELETED (replaced by tablet 1a57e0ff6b6a422aa0e69678b7bd6762) from {username='slave'} at 127.0.0.1:35560
I20250905 07:50:19.250226 2734 tablet_bootstrap.cc:654] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.251741 2748 tablet_replica.cc:331] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:19.252332 2735 tablet_bootstrap.cc:492] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.252542 2735 ts_tablet_manager.cc:1397] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250905 07:50:19.252802 2744 tablet_replica.cc:331] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:19.253280 2744 raft_consensus.cc:2241] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:19.253720 2744 raft_consensus.cc:2270] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:19.253986 2735 raft_consensus.cc:357] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.254210 2735 raft_consensus.cc:383] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.254328 2735 raft_consensus.cc:738] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.254662 2735 consensus_queue.cc:260] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.256333 2320 tablet_service.cc:1468] Processing CreateTablet for tablet 8b52a829672943da8121d121052697ab (DEFAULT_TABLE table=default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.256765 2320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b52a829672943da8121d121052697ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.258165 2748 raft_consensus.cc:2241] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:19.258610 2748 raft_consensus.cc:2270] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:19.259446 2745 tablet_replica.cc:331] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:19.259897 2745 raft_consensus.cc:2241] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:19.260298 2745 raft_consensus.cc:2270] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:19.260306 2600 tablet_service.cc:1468] Processing CreateTablet for tablet 8b52a829672943da8121d121052697ab (DEFAULT_TABLE table=default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.260736 2600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b52a829672943da8121d121052697ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.262109 2748 ts_tablet_manager.cc:1905] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:19.264662 2319 tablet_service.cc:1468] Processing CreateTablet for tablet 1a57e0ff6b6a422aa0e69678b7bd6762 (DEFAULT_TABLE table=default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.265048 2601 tablet_service.cc:1468] Processing CreateTablet for tablet 1a57e0ff6b6a422aa0e69678b7bd6762 (DEFAULT_TABLE table=default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.265563 2601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a57e0ff6b6a422aa0e69678b7bd6762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.273272 2462 tablet_service.cc:1468] Processing CreateTablet for tablet 1a57e0ff6b6a422aa0e69678b7bd6762 (DEFAULT_TABLE table=default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.273820 2462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a57e0ff6b6a422aa0e69678b7bd6762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.266471 2744 ts_tablet_manager.cc:1905] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:19.271225 2319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a57e0ff6b6a422aa0e69678b7bd6762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.275986 2734 tablet_bootstrap.cc:492] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.276163 2734 ts_tablet_manager.cc:1397] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250905 07:50:19.277166 2734 raft_consensus.cc:357] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.277369 2734 raft_consensus.cc:383] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.277431 2734 raft_consensus.cc:738] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.277674 2734 consensus_queue.cc:260] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.278154 2734 ts_tablet_manager.cc:1428] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 07:50:19.278460 2734 tablet_bootstrap.cc:492] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.280138 2734 tablet_bootstrap.cc:654] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.271883 2735 ts_tablet_manager.cc:1428] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.019s user 0.006s sys 0.005s
I20250905 07:50:19.283469 2735 tablet_bootstrap.cc:492] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.285823 2735 tablet_bootstrap.cc:654] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.298451 2745 ts_tablet_manager.cc:1905] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:19.300323 2733 tablet_bootstrap.cc:492] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.300611 2733 ts_tablet_manager.cc:1397] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.058s user 0.020s sys 0.006s
I20250905 07:50:19.303452 2733 raft_consensus.cc:357] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.303766 2733 raft_consensus.cc:383] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.303973 2733 raft_consensus.cc:738] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.304376 2733 consensus_queue.cc:260] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.306391 2733 ts_tablet_manager.cc:1428] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250905 07:50:19.307896 2733 tablet_bootstrap.cc:492] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.309747 2735 tablet_bootstrap.cc:492] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.309887 2734 tablet_bootstrap.cc:492] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.309919 2735 ts_tablet_manager.cc:1397] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250905 07:50:19.310050 2734 ts_tablet_manager.cc:1397] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.008s
I20250905 07:50:19.310380 2733 tablet_bootstrap.cc:654] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.310819 2734 raft_consensus.cc:357] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.310798 2735 raft_consensus.cc:357] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.310997 2734 raft_consensus.cc:383] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.310997 2735 raft_consensus.cc:383] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.311096 2735 raft_consensus.cc:738] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.311095 2734 raft_consensus.cc:738] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.311375 2734 consensus_queue.cc:260] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.311395 2735 consensus_queue.cc:260] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.311908 2735 ts_tablet_manager.cc:1428] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:19.312230 2735 tablet_bootstrap.cc:492] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.312284 2734 ts_tablet_manager.cc:1428] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250905 07:50:19.312629 2734 tablet_bootstrap.cc:492] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.314067 2735 tablet_bootstrap.cc:654] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.314350 2734 tablet_bootstrap.cc:654] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.322525 2744 ts_tablet_manager.cc:1918] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:19.322716 2744 log.cc:1199] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/0c1289e4420649eb99ac6102e721db0c
I20250905 07:50:19.323189 2744 ts_tablet_manager.cc:1939] T 0c1289e4420649eb99ac6102e721db0c P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:19.325275 2748 ts_tablet_manager.cc:1918] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:19.325479 2748 log.cc:1199] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/0c1289e4420649eb99ac6102e721db0c
I20250905 07:50:19.326897 2735 tablet_bootstrap.cc:492] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.327070 2735 ts_tablet_manager.cc:1397] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250905 07:50:19.327909 2745 ts_tablet_manager.cc:1918] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:19.327888 2735 raft_consensus.cc:357] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.328126 2735 raft_consensus.cc:383] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.328210 2735 raft_consensus.cc:738] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.328559 2735 consensus_queue.cc:260] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.328868 2748 ts_tablet_manager.cc:1939] T 0c1289e4420649eb99ac6102e721db0c P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:19.330837 2735 ts_tablet_manager.cc:1428] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 07:50:19.331076 2745 log.cc:1199] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/0c1289e4420649eb99ac6102e721db0c
I20250905 07:50:19.331185 2735 tablet_bootstrap.cc:492] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.331704 2745 ts_tablet_manager.cc:1939] T 0c1289e4420649eb99ac6102e721db0c P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:19.333266 2735 tablet_bootstrap.cc:654] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.335216 1996 catalog_manager.cc:4928] TS 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): tablet 0c1289e4420649eb99ac6102e721db0c (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]) successfully deleted
I20250905 07:50:19.336119 1996 catalog_manager.cc:4928] TS 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): tablet 0c1289e4420649eb99ac6102e721db0c (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]) successfully deleted
I20250905 07:50:19.336329 2734 tablet_bootstrap.cc:492] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.336521 2734 ts_tablet_manager.cc:1397] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250905 07:50:19.337373 2734 raft_consensus.cc:357] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.337579 2734 raft_consensus.cc:383] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.337560 1996 catalog_manager.cc:4928] TS b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): tablet 0c1289e4420649eb99ac6102e721db0c (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]) successfully deleted
I20250905 07:50:19.337646 2734 raft_consensus.cc:738] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.337913 2734 consensus_queue.cc:260] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.338553 2734 ts_tablet_manager.cc:1428] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:19.338898 2734 tablet_bootstrap.cc:492] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.341022 2734 tablet_bootstrap.cc:654] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.347031 2735 tablet_bootstrap.cc:492] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.347208 2735 ts_tablet_manager.cc:1397] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250905 07:50:19.348091 2735 raft_consensus.cc:357] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.348287 2735 raft_consensus.cc:383] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.348453 2735 raft_consensus.cc:738] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.348711 2735 consensus_queue.cc:260] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.352012 2735 ts_tablet_manager.cc:1428] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250905 07:50:19.355782 2734 tablet_bootstrap.cc:492] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.360105 2734 ts_tablet_manager.cc:1397] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20250905 07:50:19.361399 2734 raft_consensus.cc:357] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.361755 2734 raft_consensus.cc:383] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.361912 2734 raft_consensus.cc:738] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.362303 2734 consensus_queue.cc:260] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.364436 2734 ts_tablet_manager.cc:1428] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250905 07:50:19.366992 2004 catalog_manager.cc:2149] created HMS catalog entry for table [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:19.375547 2004 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c1289e4420649eb99ac6102e721db0c from {username='slave'} at 127.0.0.1:35886
I20250905 07:50:19.377986 2009 master_service.cc:948] ReplaceTablet: received request to replace tablet f6d24d267ac142fca1cec34552e395fd from {username='slave'} at 127.0.0.1:35886
I20250905 07:50:19.378257 2741 raft_consensus.cc:491] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:19.378538 2741 raft_consensus.cc:513] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.380219 2733 tablet_bootstrap.cc:492] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.380416 2733 ts_tablet_manager.cc:1397] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.073s user 0.030s sys 0.037s
I20250905 07:50:19.380504 2741 leader_election.cc:290] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.381242 2733 raft_consensus.cc:357] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.381455 2733 raft_consensus.cc:383] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.381538 2733 raft_consensus.cc:738] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.381839 2733 consensus_queue.cc:260] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.382354 2733 ts_tablet_manager.cc:1428] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:19.383024 2733 tablet_bootstrap.cc:492] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.386197 2733 tablet_bootstrap.cc:654] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.399065 2462 tablet_service.cc:1468] Processing CreateTablet for tablet 09612f5ac871434399ea0cc0a7060c7b (DEFAULT_TABLE table=default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.399688 2462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09612f5ac871434399ea0cc0a7060c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.404671 2319 tablet_service.cc:1468] Processing CreateTablet for tablet 09612f5ac871434399ea0cc0a7060c7b (DEFAULT_TABLE table=default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.405265 2319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09612f5ac871434399ea0cc0a7060c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.416989 2009 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet f6d24d267ac142fca1cec34552e395fd
I20250905 07:50:19.417552 2009 catalog_manager.cc:6652] ReplaceTablet: tablet f6d24d267ac142fca1cec34552e395fd deleted and replaced by tablet 083c74057a6945288b259678e0ff2223
I20250905 07:50:19.418931 2009 master_service.cc:948] ReplaceTablet: received request to replace tablet f6d24d267ac142fca1cec34552e395fd from {username='slave'} at 127.0.0.1:35886
I20250905 07:50:19.419384 2009 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c1289e4420649eb99ac6102e721db0c from {username='slave'} at 127.0.0.1:35886
I20250905 07:50:19.423784 2461 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d24d267ac142fca1cec34552e395fd with delete_type TABLET_DATA_DELETED (replaced by tablet 083c74057a6945288b259678e0ff2223) from {username='slave'} at 127.0.0.1:52248
I20250905 07:50:19.424124 2320 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d24d267ac142fca1cec34552e395fd with delete_type TABLET_DATA_DELETED (replaced by tablet 083c74057a6945288b259678e0ff2223) from {username='slave'} at 127.0.0.1:52948
I20250905 07:50:19.424154 2745 tablet_replica.cc:331] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:19.424398 2745 raft_consensus.cc:2241] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:19.424598 2745 raft_consensus.cc:2270] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:19.424896 2744 tablet_replica.cc:331] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:19.425109 2744 raft_consensus.cc:2241] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:19.425290 2744 raft_consensus.cc:2270] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:19.425452 2745 ts_tablet_manager.cc:1905] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 07:50:19.429119 2669 tablet.cc:2378] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:50:19.432969 2744 ts_tablet_manager.cc:1905] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:19.419876 2009 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "default.Table_a908ec5ae1aa46e8b872ff681d610476" } new_table_name: "default.Table_23be441c886b4e17929155f056fdb23b" modify_external_catalogs: true
I20250905 07:50:19.439400 2744 ts_tablet_manager.cc:1918] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:19.439581 2744 log.cc:1199] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/f6d24d267ac142fca1cec34552e395fd
I20250905 07:50:19.439993 2744 ts_tablet_manager.cc:1939] T f6d24d267ac142fca1cec34552e395fd P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:19.441141 1996 catalog_manager.cc:4928] TS b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): tablet f6d24d267ac142fca1cec34552e395fd (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]) successfully deleted
I20250905 07:50:19.442716 2745 ts_tablet_manager.cc:1918] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:19.442883 2745 log.cc:1199] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/f6d24d267ac142fca1cec34552e395fd
I20250905 07:50:19.443234 2745 ts_tablet_manager.cc:1939] T f6d24d267ac142fca1cec34552e395fd P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:19.444329 1996 catalog_manager.cc:4928] TS 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): tablet f6d24d267ac142fca1cec34552e395fd (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]) successfully deleted
I20250905 07:50:19.450585 2004 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_06530af7099e45c7b150408f8c1a3436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:19.451653 2320 tablet_service.cc:1468] Processing CreateTablet for tablet 083c74057a6945288b259678e0ff2223 (DEFAULT_TABLE table=default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.452203 2320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 083c74057a6945288b259678e0ff2223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.456804 2461 tablet_service.cc:1468] Processing CreateTablet for tablet 083c74057a6945288b259678e0ff2223 (DEFAULT_TABLE table=default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.457193 2461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 083c74057a6945288b259678e0ff2223. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 07:50:19.459856 2531 tablet.cc:2378] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:50:19.462114 2388 tablet.cc:2378] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 07:50:19.470009 2735 tablet_bootstrap.cc:492] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.472522 2735 tablet_bootstrap.cc:654] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.490638 2735 tablet_bootstrap.cc:492] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.490908 2735 ts_tablet_manager.cc:1397] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.003s
I20250905 07:50:19.492210 2735 raft_consensus.cc:357] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.492606 2735 raft_consensus.cc:383] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.492720 2735 raft_consensus.cc:738] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.494603 2735 consensus_queue.cc:260] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.500717 2735 ts_tablet_manager.cc:1428] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20250905 07:50:19.505129 2005 catalog_manager.cc:2149] created HMS catalog entry for table [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:19.526603 2601 tablet_service.cc:1468] Processing CreateTablet for tablet 09612f5ac871434399ea0cc0a7060c7b (DEFAULT_TABLE table=default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.527276 2601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09612f5ac871434399ea0cc0a7060c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.537503 2734 tablet_bootstrap.cc:492] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.539456 2734 tablet_bootstrap.cc:654] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.539659 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b52a829672943da8121d121052697ab" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:19.540086 2600 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d24d267ac142fca1cec34552e395fd with delete_type TABLET_DATA_DELETED (replaced by tablet 083c74057a6945288b259678e0ff2223) from {username='slave'} at 127.0.0.1:35560
I20250905 07:50:19.540200 2483 raft_consensus.cc:2466] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.540673 2599 tablet_service.cc:1468] Processing CreateTablet for tablet 083c74057a6945288b259678e0ff2223 (DEFAULT_TABLE table=default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.541278 2599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 083c74057a6945288b259678e0ff2223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.541810 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b52a829672943da8121d121052697ab" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:19.542289 2340 raft_consensus.cc:2466] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.542783 2748 tablet_replica.cc:331] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:19.542984 2748 raft_consensus.cc:2241] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:19.543160 2748 raft_consensus.cc:2270] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:19.544258 2748 ts_tablet_manager.cc:1905] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:19.541275 2552 leader_election.cc:304] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:19.553398 2748 ts_tablet_manager.cc:1918] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:19.555115 2748 log.cc:1199] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/f6d24d267ac142fca1cec34552e395fd
I20250905 07:50:19.555738 2748 ts_tablet_manager.cc:1939] T f6d24d267ac142fca1cec34552e395fd P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:19.555816 2005 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_6abbacd07c2f4ab4b74761cd100dcf58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:19.557252 2741 raft_consensus.cc:2802] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:19.557466 2741 raft_consensus.cc:491] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:19.557580 2741 raft_consensus.cc:3058] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.558039 2735 tablet_bootstrap.cc:492] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.559834 2735 tablet_bootstrap.cc:654] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.562755 2741 raft_consensus.cc:513] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.566592 2733 tablet_bootstrap.cc:492] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.566817 2733 ts_tablet_manager.cc:1397] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.184s user 0.073s sys 0.044s
I20250905 07:50:19.567689 2733 raft_consensus.cc:357] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.567888 2733 raft_consensus.cc:383] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.568003 2733 raft_consensus.cc:738] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.568306 2733 consensus_queue.cc:260] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.568864 2733 ts_tablet_manager.cc:1428] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:19.569168 2733 tablet_bootstrap.cc:492] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.566651 2741 leader_election.cc:290] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Requested vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.570991 2733 tablet_bootstrap.cc:654] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.571981 2319 tablet_service.cc:1468] Processing CreateTablet for tablet 3743b39493ed452e89b1e3223bb1ba88 (DEFAULT_TABLE table=default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.572631 2319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3743b39493ed452e89b1e3223bb1ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.567510 2734 tablet_bootstrap.cc:492] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.573671 2734 ts_tablet_manager.cc:1397] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.005s
I20250905 07:50:19.574723 2734 raft_consensus.cc:357] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.574936 2734 raft_consensus.cc:383] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.575009 2734 raft_consensus.cc:738] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.575309 2734 consensus_queue.cc:260] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.576872 2733 tablet_bootstrap.cc:492] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.576908 2734 ts_tablet_manager.cc:1428] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 07:50:19.577059 2733 ts_tablet_manager.cc:1397] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20250905 07:50:19.577198 2734 tablet_bootstrap.cc:492] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.577857 2733 raft_consensus.cc:357] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.578092 2733 raft_consensus.cc:383] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.578171 2733 raft_consensus.cc:738] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.578974 2734 tablet_bootstrap.cc:654] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.577917 2462 tablet_service.cc:1468] Processing CreateTablet for tablet 3743b39493ed452e89b1e3223bb1ba88 (DEFAULT_TABLE table=default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.579840 2462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3743b39493ed452e89b1e3223bb1ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.580199 2733 consensus_queue.cc:260] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.584704 2733 ts_tablet_manager.cc:1428] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250905 07:50:19.588371 2733 tablet_bootstrap.cc:492] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.591277 2733 tablet_bootstrap.cc:654] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
W20250905 07:50:19.591689 2552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.178.130:37221, user_credentials={real_user=slave}} blocked reactor thread for 50542.4us
I20250905 07:50:19.595767 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b52a829672943da8121d121052697ab" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:19.595979 2735 tablet_bootstrap.cc:492] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.596139 2735 ts_tablet_manager.cc:1397] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.005s
I20250905 07:50:19.596081 2340 raft_consensus.cc:3058] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.597114 2735 raft_consensus.cc:357] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.597306 2735 raft_consensus.cc:383] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.597394 2735 raft_consensus.cc:738] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.597672 2735 consensus_queue.cc:260] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.600841 2340 raft_consensus.cc:2466] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.601883 2735 ts_tablet_manager.cc:1428] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250905 07:50:19.602015 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b52a829672943da8121d121052697ab" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:19.602277 2483 raft_consensus.cc:3058] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.605262 2733 tablet_bootstrap.cc:492] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.605451 2733 ts_tablet_manager.cc:1397] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20250905 07:50:19.605998 2483 raft_consensus.cc:2466] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.606454 2741 raft_consensus.cc:491] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:19.606484 2733 raft_consensus.cc:357] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.606686 2733 raft_consensus.cc:383] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.606662 2741 raft_consensus.cc:513] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.606787 2733 raft_consensus.cc:738] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.605237 2552 leader_election.cc:304] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:19.607005 2733 consensus_queue.cc:260] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.607544 2733 ts_tablet_manager.cc:1428] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 07:50:19.610642 2733 tablet_bootstrap.cc:492] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.612561 2733 tablet_bootstrap.cc:654] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.612972 1996 catalog_manager.cc:4928] TS 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): tablet f6d24d267ac142fca1cec34552e395fd (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]) successfully deleted
I20250905 07:50:19.607363 2741 leader_election.cc:290] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:19.613480 2741 raft_consensus.cc:2802] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:19.614141 2599 tablet_service.cc:1468] Processing CreateTablet for tablet 3743b39493ed452e89b1e3223bb1ba88 (DEFAULT_TABLE table=default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.614717 2599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3743b39493ed452e89b1e3223bb1ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.615712 2735 tablet_bootstrap.cc:492] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.615999 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09612f5ac871434399ea0cc0a7060c7b" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:19.616300 2340 raft_consensus.cc:2466] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.619237 2735 tablet_bootstrap.cc:654] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.613843 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09612f5ac871434399ea0cc0a7060c7b" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:19.620906 2483 raft_consensus.cc:2466] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.621513 2552 leader_election.cc:304] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:19.621565 2734 tablet_bootstrap.cc:492] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.621822 2734 ts_tablet_manager.cc:1397] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.004s
I20250905 07:50:19.622690 2734 raft_consensus.cc:357] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.622882 2734 raft_consensus.cc:383] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.622972 2734 raft_consensus.cc:738] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.624468 2734 consensus_queue.cc:260] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.632526 2735 tablet_bootstrap.cc:492] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.632794 2735 ts_tablet_manager.cc:1397] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250905 07:50:19.635738 2772 raft_consensus.cc:2802] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:19.635898 2772 raft_consensus.cc:491] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:19.637583 2772 raft_consensus.cc:3058] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.639156 2741 raft_consensus.cc:695] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Becoming Leader. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Running, Role: LEADER
I20250905 07:50:19.639618 2741 consensus_queue.cc:237] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.640359 2734 ts_tablet_manager.cc:1428] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250905 07:50:19.641767 2735 raft_consensus.cc:357] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.641999 2735 raft_consensus.cc:383] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.642077 2735 raft_consensus.cc:738] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.647316 2734 tablet_bootstrap.cc:492] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.649384 2734 tablet_bootstrap.cc:654] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.652120 2733 tablet_bootstrap.cc:492] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.652403 2733 ts_tablet_manager.cc:1397] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.016s
I20250905 07:50:19.652302 2735 consensus_queue.cc:260] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.653379 2733 raft_consensus.cc:357] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.653650 2733 raft_consensus.cc:383] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.653770 2733 raft_consensus.cc:738] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.654090 2733 consensus_queue.cc:260] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.655684 2733 ts_tablet_manager.cc:1428] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:50:19.656654 2735 ts_tablet_manager.cc:1428] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.024s user 0.000s sys 0.003s
I20250905 07:50:19.658519 2772 raft_consensus.cc:513] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.665227 2772 leader_election.cc:290] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Requested vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:19.665729 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09612f5ac871434399ea0cc0a7060c7b" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:19.665882 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09612f5ac871434399ea0cc0a7060c7b" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:19.665982 2483 raft_consensus.cc:3058] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.666185 2340 raft_consensus.cc:3058] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.668804 2483 raft_consensus.cc:2466] T 09612f5ac871434399ea0cc0a7060c7b P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.669008 2734 tablet_bootstrap.cc:492] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.669289 2734 ts_tablet_manager.cc:1397] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250905 07:50:19.670253 2734 raft_consensus.cc:357] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.670481 2734 raft_consensus.cc:383] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.670572 2734 raft_consensus.cc:738] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.670856 2734 consensus_queue.cc:260] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.671391 2734 ts_tablet_manager.cc:1428] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 07:50:19.676174 2552 leader_election.cc:304] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:19.676625 2741 raft_consensus.cc:491] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:19.676795 2741 raft_consensus.cc:513] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.677368 2741 leader_election.cc:290] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.677523 2741 raft_consensus.cc:2802] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:19.677664 2741 raft_consensus.cc:695] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Becoming Leader. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Running, Role: LEADER
I20250905 07:50:19.682655 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3743b39493ed452e89b1e3223bb1ba88" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:19.682914 2483 raft_consensus.cc:2466] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.683568 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3743b39493ed452e89b1e3223bb1ba88" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:19.683861 2339 raft_consensus.cc:2466] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.686252 2552 leader_election.cc:304] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:19.686878 2770 raft_consensus.cc:2802] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:19.687013 2770 raft_consensus.cc:491] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:19.687146 2770 raft_consensus.cc:3058] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.688117 2741 consensus_queue.cc:237] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.692224 2770 raft_consensus.cc:513] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.692831 2770 leader_election.cc:290] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Requested vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.693663 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3743b39493ed452e89b1e3223bb1ba88" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:19.693884 2483 raft_consensus.cc:3058] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.696476 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3743b39493ed452e89b1e3223bb1ba88" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:19.696761 2339 raft_consensus.cc:3058] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.701650 2483 raft_consensus.cc:2466] T 3743b39493ed452e89b1e3223bb1ba88 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.702229 2552 leader_election.cc:304] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:19.702663 2772 raft_consensus.cc:2802] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:19.702817 2772 raft_consensus.cc:695] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Becoming Leader. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Running, Role: LEADER
I20250905 07:50:19.703112 2772 consensus_queue.cc:237] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.704320 2340 raft_consensus.cc:2466] T 09612f5ac871434399ea0cc0a7060c7b P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.705196 2339 raft_consensus.cc:2466] T 3743b39493ed452e89b1e3223bb1ba88 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.712543 2750 raft_consensus.cc:491] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:19.712769 2750 raft_consensus.cc:513] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.714845 2750 leader_election.cc:290] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.736222 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9984876006444aa460afd1604eb359" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:19.736531 2483 raft_consensus.cc:2466] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:19.737110 2621 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9984876006444aa460afd1604eb359" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:19.737259 2273 leader_election.cc:304] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:19.737541 2621 raft_consensus.cc:2466] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:19.737900 2750 raft_consensus.cc:2802] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:19.738050 2750 raft_consensus.cc:491] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:19.738159 2750 raft_consensus.cc:3058] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.741047 2750 raft_consensus.cc:513] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:19.741734 2750 leader_election.cc:290] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Requested vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.742591 2621 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9984876006444aa460afd1604eb359" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:19.742792 2621 raft_consensus.cc:3058] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.745915 2621 raft_consensus.cc:2466] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:19.746889 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9984876006444aa460afd1604eb359" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:19.747125 2483 raft_consensus.cc:3058] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.750022 2483 raft_consensus.cc:2466] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:19.751107 2273 leader_election.cc:304] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:19.751773 2750 raft_consensus.cc:2802] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:19.753592 2750 raft_consensus.cc:695] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: Becoming Leader. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Running, Role: LEADER
I20250905 07:50:19.754263 2750 consensus_queue.cc:237] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
07:50:19.840 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_cc7e423f936848e1b8abbb22c3440719 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250905 07:50:19.843103 2007 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=fc68c5780acc42b280c33bd899c74b07]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_cc7e423f936848e1b8abbb22c3440719 already exists)
I20250905 07:50:19.843640 2007 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35886 (request call id 18) took 1011 ms. Trace:
I20250905 07:50:19.843791 2007 rpcz_store.cc:276] 0905 07:50:18.832556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 07:50:18.833235 (+ 679us) service_pool.cc:225] Handling call
0905 07:50:18.855004 (+ 21769us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 07:50:18.855444 (+ 440us) catalog_manager.cc:2122] Created new table and tablet info
0905 07:50:19.843575 (+988131us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":649201,"Hive Metastore.run_cpu_time_us":1029,"Hive Metastore.run_wall_time_us":338080}
I20250905 07:50:19.850306 2007 catalog_manager.cc:5582] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to b902ef48db944a02813b8543b2f91a7e (127.1.178.129). New cstate: current_term: 1 leader_uuid: "b902ef48db944a02813b8543b2f91a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } health_report { overall_health: UNKNOWN } } }
I20250905 07:50:19.867451 2007 catalog_manager.cc:5582] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131). New cstate: current_term: 1 leader_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } health_report { overall_health: HEALTHY } } }
I20250905 07:50:19.868867 2007 catalog_manager.cc:5582] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131). New cstate: current_term: 1 leader_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } health_report { overall_health: HEALTHY } } }
I20250905 07:50:19.869692 2007 catalog_manager.cc:5582] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131). New cstate: current_term: 1 leader_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } health_report { overall_health: HEALTHY } } }
I20250905 07:50:19.879040 2007 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_8f62349a373e45b8ae063dafbfc3cdcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:19.881529 2008 catalog_manager.cc:2149] created HMS catalog entry for table [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:19.887315 2008 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35886 (request call id 17) took 1054 ms. Trace:
I20250905 07:50:19.887463 2008 rpcz_store.cc:276] 0905 07:50:18.832417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 07:50:18.832494 (+ 77us) service_pool.cc:225] Handling call
0905 07:50:18.856003 (+ 23509us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 07:50:18.856431 (+ 428us) catalog_manager.cc:2122] Created new table and tablet info
0905 07:50:19.881469 (+1025038us) catalog_manager.cc:2148] Created new table in HMS catalog
0905 07:50:19.887013 (+ 5544us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 07:50:19.887157 (+ 144us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 07:50:19.887274 (+ 117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 07:50:19.882315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 07:50:19.882437 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0905 07:50:19.882443 (+ 6us) write_op.cc:435] Acquired schema lock
0905 07:50:19.882445 (+ 2us) tablet.cc:660] Decoding operations
0905 07:50:19.882549 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0905 07:50:19.882561 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0905 07:50:19.882566 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 07:50:19.882642 (+ 76us) tablet.cc:699] Row locks acquired
0905 07:50:19.882644 (+ 2us) write_op.cc:260] PREPARE: finished
0905 07:50:19.882679 (+ 35us) write_op.cc:270] Start()
0905 07:50:19.882707 (+ 28us) write_op.cc:276] Timestamp: P: 1757058619882637 usec, L: 0
0905 07:50:19.882710 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 07:50:19.882867 (+ 157us) log.cc:844] Serialized 432 byte log entry
0905 07:50:19.886274 (+ 3407us) op_driver.cc:464] REPLICATION: finished
0905 07:50:19.886426 (+ 152us) write_op.cc:301] APPLY: starting
0905 07:50:19.886443 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 07:50:19.886488 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0905 07:50:19.886522 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 07:50:19.886627 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 07:50:19.886667 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 07:50:19.886674 (+ 7us) write_op.cc:312] APPLY: finished
0905 07:50:19.886749 (+ 75us) log.cc:844] Serialized 28 byte log entry
0905 07:50:19.886807 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0905 07:50:19.886825 (+ 18us) write_op.cc:454] Released schema lock
0905 07:50:19.886830 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 07:50:19.886914 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0905 07:50:19.886920 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":986388,"Hive Metastore.run_cpu_time_us":351,"Hive Metastore.run_wall_time_us":38259,"spinlock_wait_cycles":29568,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":465,"num_ops":2,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":660,"raft.queue_time_us":159,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":282,"replication_time_us":3536}]]}
I20250905 07:50:19.888635 2008 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "default.Table_fbd6c4d89fd54fdd9765576cc3cf5cbc" } new_table_name: "default.Table_68b9ab7fcb3245ad84e0fe0e5c05d746" modify_external_catalogs: true
W20250905 07:50:19.893898 2751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:19.894387 2751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:19.894526 2751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250905 07:50:19.898595 2319 tablet_service.cc:1468] Processing CreateTablet for tablet 9e2858d296ec4022a337e6f3d6fa629d (DEFAULT_TABLE table=default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.899139 2319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2858d296ec4022a337e6f3d6fa629d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.900123 2462 tablet_service.cc:1468] Processing CreateTablet for tablet 9e2858d296ec4022a337e6f3d6fa629d (DEFAULT_TABLE table=default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.900533 2462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2858d296ec4022a337e6f3d6fa629d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.904692 2599 tablet_service.cc:1468] Processing CreateTablet for tablet 9e2858d296ec4022a337e6f3d6fa629d (DEFAULT_TABLE table=default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.905102 2599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2858d296ec4022a337e6f3d6fa629d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.905347 2735 tablet_bootstrap.cc:492] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.907063 2735 tablet_bootstrap.cc:654] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
W20250905 07:50:19.908288 2751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:19.908411 2751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:19.908509 2751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:19.908588 2751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:19.916440 2734 tablet_bootstrap.cc:492] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:19.918812 2734 tablet_bootstrap.cc:654] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.920034 2006 catalog_manager.cc:2149] created HMS catalog entry for table [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:19.921515 2741 raft_consensus.cc:491] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:19.921682 2741 raft_consensus.cc:513] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.922344 2741 leader_election.cc:290] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.923362 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "083c74057a6945288b259678e0ff2223" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:19.923609 2339 raft_consensus.cc:2466] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.924458 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "083c74057a6945288b259678e0ff2223" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:19.924664 2483 raft_consensus.cc:2466] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:19.925526 2751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.188:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.178.188
--webserver_port=42041
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:19.925671 2552 leader_election.cc:304] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:19.926472 2741 raft_consensus.cc:2802] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:19.926592 2741 raft_consensus.cc:491] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:19.926685 2741 raft_consensus.cc:3058] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.926734 2006 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35886 (request call id 19) took 1094 ms. Trace:
I20250905 07:50:19.926851 2006 rpcz_store.cc:276] 0905 07:50:18.832684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 07:50:18.832728 (+ 44us) service_pool.cc:225] Handling call
0905 07:50:18.856899 (+ 24171us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 07:50:18.857363 (+ 464us) catalog_manager.cc:2122] Created new table and tablet info
0905 07:50:19.919971 (+1062608us) catalog_manager.cc:2148] Created new table in HMS catalog
0905 07:50:19.926413 (+ 6442us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 07:50:19.926545 (+ 132us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 07:50:19.926688 (+ 143us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 07:50:19.920850 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 07:50:19.921029 (+ 179us) write_op.cc:432] Acquiring schema lock in shared mode
0905 07:50:19.921039 (+ 10us) write_op.cc:435] Acquired schema lock
0905 07:50:19.921044 (+ 5us) tablet.cc:660] Decoding operations
0905 07:50:19.921187 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
0905 07:50:19.921205 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0905 07:50:19.921212 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 07:50:19.921334 (+ 122us) tablet.cc:699] Row locks acquired
0905 07:50:19.921338 (+ 4us) write_op.cc:260] PREPARE: finished
0905 07:50:19.921403 (+ 65us) write_op.cc:270] Start()
0905 07:50:19.921443 (+ 40us) write_op.cc:276] Timestamp: P: 1757058619921352 usec, L: 0
0905 07:50:19.921447 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 07:50:19.921655 (+ 208us) log.cc:844] Serialized 432 byte log entry
0905 07:50:19.925660 (+ 4005us) op_driver.cc:464] REPLICATION: finished
0905 07:50:19.925803 (+ 143us) write_op.cc:301] APPLY: starting
0905 07:50:19.925818 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0905 07:50:19.925869 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0905 07:50:19.925895 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 07:50:19.926001 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 07:50:19.926038 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 07:50:19.926045 (+ 7us) write_op.cc:312] APPLY: finished
0905 07:50:19.926141 (+ 96us) log.cc:844] Serialized 28 byte log entry
0905 07:50:19.926213 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0905 07:50:19.926232 (+ 19us) write_op.cc:454] Released schema lock
0905 07:50:19.926239 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 07:50:19.926305 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 07:50:19.926308 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1023776,"Hive Metastore.run_cpu_time_us":284,"Hive Metastore.run_wall_time_us":38638,"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":493,"apply.run_wall_time_us":493,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":932,"prepare.run_wall_time_us":933,"raft.queue_time_us":980,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":305,"replication_time_us":4178}]]}
I20250905 07:50:19.927997 2751 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:19.929481 2741 raft_consensus.cc:513] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.930446 2751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 07:50:19.930801 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "083c74057a6945288b259678e0ff2223" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:19.931033 2339 raft_consensus.cc:3058] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.927814 2006 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "default.Table_7c0359e055804d0b9a84bd4dd5e60634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:19.933769 2339 raft_consensus.cc:2466] T 083c74057a6945288b259678e0ff2223 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.935384 2552 leader_election.cc:304] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:19.935727 2772 raft_consensus.cc:2802] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:19.935863 2772 raft_consensus.cc:695] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Becoming Leader. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Running, Role: LEADER
I20250905 07:50:19.936159 2772 consensus_queue.cc:237] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.940001 2461 tablet_service.cc:1468] Processing CreateTablet for tablet 0c28568a8557410c9104026b8f1cf3f1 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.940474 2461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c28568a8557410c9104026b8f1cf3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.944168 2733 tablet_bootstrap.cc:492] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:19.944058 2735 tablet_bootstrap.cc:492] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:19.944842 2735 ts_tablet_manager.cc:1397] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.009s
I20250905 07:50:19.945709 2735 raft_consensus.cc:357] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.945928 2733 tablet_bootstrap.cc:654] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:19.945911 2735 raft_consensus.cc:383] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.946012 2735 raft_consensus.cc:738] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.946295 2735 consensus_queue.cc:260] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.947109 2735 ts_tablet_manager.cc:1428] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:19.955726 2734 tablet_bootstrap.cc:492] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:19.956012 2734 ts_tablet_manager.cc:1397] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20250905 07:50:19.957001 2734 raft_consensus.cc:357] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.958381 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "083c74057a6945288b259678e0ff2223" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:19.958568 2483 raft_consensus.cc:3058] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:19.962045 2483 raft_consensus.cc:2466] T 083c74057a6945288b259678e0ff2223 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:19.963707 2319 tablet_service.cc:1468] Processing CreateTablet for tablet 0c28568a8557410c9104026b8f1cf3f1 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.964251 2319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c28568a8557410c9104026b8f1cf3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.972069 2734 raft_consensus.cc:383] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:19.972258 2734 raft_consensus.cc:738] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:19.972600 2734 consensus_queue.cc:260] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:19.974256 2734 ts_tablet_manager.cc:1428] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 07:50:19.957744 2741 leader_election.cc:290] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Requested vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:19.977458 2599 tablet_service.cc:1468] Processing CreateTablet for tablet 0c28568a8557410c9104026b8f1cf3f1 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:19.977916 2599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c28568a8557410c9104026b8f1cf3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:19.996753 2733 tablet_bootstrap.cc:492] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:19.997148 2733 ts_tablet_manager.cc:1397] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.009s
I20250905 07:50:19.997553 2735 tablet_bootstrap.cc:492] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:19.999402 2735 tablet_bootstrap.cc:654] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.001655 2735 tablet_bootstrap.cc:492] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:20.001837 2735 ts_tablet_manager.cc:1397] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250905 07:50:20.002605 2735 raft_consensus.cc:357] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.002820 2735 raft_consensus.cc:383] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.002888 2735 raft_consensus.cc:738] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.003115 2735 consensus_queue.cc:260] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.005373 2735 ts_tablet_manager.cc:1428] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 07:50:20.009668 2733 raft_consensus.cc:357] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.011696 2733 raft_consensus.cc:383] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.011874 2733 raft_consensus.cc:738] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.012204 2733 consensus_queue.cc:260] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.020030 2734 tablet_bootstrap.cc:492] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:20.021737 2734 tablet_bootstrap.cc:654] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.023304 2733 ts_tablet_manager.cc:1428] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250905 07:50:20.023938 2012 catalog_manager.cc:2149] created HMS catalog entry for table [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:20.024179 2734 tablet_bootstrap.cc:492] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:20.024334 2734 ts_tablet_manager.cc:1397] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250905 07:50:20.025086 2734 raft_consensus.cc:357] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.025300 2734 raft_consensus.cc:383] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.025393 2734 raft_consensus.cc:738] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.025677 2734 consensus_queue.cc:260] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.026154 2734 ts_tablet_manager.cc:1428] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:20.030351 2012 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35886 (request call id 13) took 1198 ms. Trace:
I20250905 07:50:20.030510 2012 rpcz_store.cc:276] 0905 07:50:18.831875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 07:50:18.833107 (+ 1232us) service_pool.cc:225] Handling call
0905 07:50:18.857824 (+ 24717us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 07:50:18.858297 (+ 473us) catalog_manager.cc:2122] Created new table and tablet info
0905 07:50:20.023868 (+1165571us) catalog_manager.cc:2148] Created new table in HMS catalog
0905 07:50:20.030082 (+ 6214us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 07:50:20.030197 (+ 115us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 07:50:20.030308 (+ 111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 07:50:20.024786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 07:50:20.024907 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0905 07:50:20.024914 (+ 7us) write_op.cc:435] Acquired schema lock
0905 07:50:20.024916 (+ 2us) tablet.cc:660] Decoding operations
0905 07:50:20.025018 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0905 07:50:20.025033 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 07:50:20.025039 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0905 07:50:20.025122 (+ 83us) tablet.cc:699] Row locks acquired
0905 07:50:20.025125 (+ 3us) write_op.cc:260] PREPARE: finished
0905 07:50:20.025169 (+ 44us) write_op.cc:270] Start()
0905 07:50:20.025205 (+ 36us) write_op.cc:276] Timestamp: P: 1757058620025113 usec, L: 0
0905 07:50:20.025209 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 07:50:20.025434 (+ 225us) log.cc:844] Serialized 432 byte log entry
0905 07:50:20.029242 (+ 3808us) op_driver.cc:464] REPLICATION: finished
0905 07:50:20.029392 (+ 150us) write_op.cc:301] APPLY: starting
0905 07:50:20.029409 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 07:50:20.029459 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0905 07:50:20.029487 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 07:50:20.029644 (+ 157us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 07:50:20.029682 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 07:50:20.029689 (+ 7us) write_op.cc:312] APPLY: finished
0905 07:50:20.029774 (+ 85us) log.cc:844] Serialized 28 byte log entry
0905 07:50:20.029850 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0905 07:50:20.029872 (+ 22us) write_op.cc:454] Released schema lock
0905 07:50:20.029881 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0905 07:50:20.029961 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0905 07:50:20.029964 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1061532,"Hive Metastore.run_cpu_time_us":299,"Hive Metastore.run_wall_time_us":103685,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":559,"apply.run_wall_time_us":559,"num_ops":2,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":783,"prepare.run_wall_time_us":783,"raft.queue_time_us":945,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":255,"replication_time_us":4004}]]}
I20250905 07:50:20.031193 2733 tablet_bootstrap.cc:492] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:20.033020 2733 tablet_bootstrap.cc:654] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.033396 2012 catalog_manager.cc:5582] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131). New cstate: current_term: 1 leader_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } health_report { overall_health: HEALTHY } } }
W20250905 07:50:20.040915 2788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
F20250905 07:50:20.042886 2224 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
I20250905 07:50:20.043380 2733 tablet_bootstrap.cc:492] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
*** Check failure stack trace: ***
I20250905 07:50:20.043565 2733 ts_tablet_manager.cc:1397] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20250905 07:50:20.044448 2733 raft_consensus.cc:357] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.044656 2733 raft_consensus.cc:383] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.044730 2733 raft_consensus.cc:738] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.044996 2733 consensus_queue.cc:260] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
@ 0x7f7239789dcd google::LogMessage::Fail() at ??:0
@ 0x7f723978db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f72397897cc google::LogMessage::Flush() at ??:0
@ 0x7f7239789a11 google::LogMessage::~LogMessage() at ??:0
I20250905 07:50:20.055106 2733 ts_tablet_manager.cc:1428] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
@ 0x7f724fc057b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
W20250905 07:50:20.064849 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:20.064787 2461 tablet_service.cc:1468] Processing CreateTablet for tablet 7a4ef48a5e67406297cf21413236f563 (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:20.065239 2461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4ef48a5e67406297cf21413236f563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:20.068508 2319 tablet_service.cc:1468] Processing CreateTablet for tablet 7a4ef48a5e67406297cf21413236f563 (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:20.068964 2319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4ef48a5e67406297cf21413236f563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:20.052078 2012 master_service.cc:948] ReplaceTablet: received request to replace tablet af9984876006444aa460afd1604eb359 from {username='slave'} at 127.0.0.1:35886
I20250905 07:50:20.075381 2599 tablet_service.cc:1468] Processing CreateTablet for tablet 7a4ef48a5e67406297cf21413236f563 (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:20.075778 2599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4ef48a5e67406297cf21413236f563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:20.079360 2740 raft_consensus.cc:491] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:20.079516 2740 raft_consensus.cc:513] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:20.080147 2733 tablet_bootstrap.cc:492] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
@ 0x7f724fc04061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250905 07:50:20.081434 2740 leader_election.cc:290] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.081933 2733 tablet_bootstrap.cc:654] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.085263 2734 tablet_bootstrap.cc:492] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
@ 0x7f724fc0bcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250905 07:50:20.086993 2740 raft_consensus.cc:491] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:20.087286 2734 tablet_bootstrap.cc:654] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.087198 2740 raft_consensus.cc:513] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.087908 2740 leader_election.cc:290] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.088418 2735 tablet_bootstrap.cc:492] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:20.090312 2735 tablet_bootstrap.cc:654] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.091148 2012 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet af9984876006444aa460afd1604eb359
I20250905 07:50:20.091676 2012 catalog_manager.cc:6652] ReplaceTablet: tablet af9984876006444aa460afd1604eb359 deleted and replaced by tablet 84726ef69a6742a9b046a8833e406590
I20250905 07:50:20.092478 2462 tablet_service.cc:1515] Processing DeleteTablet for tablet af9984876006444aa460afd1604eb359 with delete_type TABLET_DATA_DELETED (replaced by tablet 84726ef69a6742a9b046a8833e406590) from {username='slave'} at 127.0.0.1:52248
I20250905 07:50:20.092741 2599 tablet_service.cc:1515] Processing DeleteTablet for tablet af9984876006444aa460afd1604eb359 with delete_type TABLET_DATA_DELETED (replaced by tablet 84726ef69a6742a9b046a8833e406590) from {username='slave'} at 127.0.0.1:35560
I20250905 07:50:20.093037 2748 tablet_replica.cc:331] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:20.093297 2748 raft_consensus.cc:2241] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:20.093494 2748 raft_consensus.cc:2270] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:20.094034 2748 ts_tablet_manager.cc:1905] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:20.103683 2463 tablet_service.cc:1468] Processing CreateTablet for tablet 84726ef69a6742a9b046a8833e406590 (DEFAULT_TABLE table=default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:20.103786 2320 tablet_service.cc:1468] Processing CreateTablet for tablet 84726ef69a6742a9b046a8833e406590 (DEFAULT_TABLE table=default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:20.104128 2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84726ef69a6742a9b046a8833e406590. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:20.104189 2320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84726ef69a6742a9b046a8833e406590. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:20.104960 2733 tablet_bootstrap.cc:492] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:20.105139 2733 ts_tablet_manager.cc:1397] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.004s
I20250905 07:50:20.106050 2733 raft_consensus.cc:357] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.106460 2319 tablet_service.cc:1515] Processing DeleteTablet for tablet af9984876006444aa460afd1604eb359 with delete_type TABLET_DATA_DELETED (replaced by tablet 84726ef69a6742a9b046a8833e406590) from {username='slave'} at 127.0.0.1:52948
I20250905 07:50:20.107635 2748 ts_tablet_manager.cc:1918] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:20.107774 2748 log.cc:1199] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/af9984876006444aa460afd1604eb359
I20250905 07:50:20.108295 2748 ts_tablet_manager.cc:1939] T af9984876006444aa460afd1604eb359 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:20.109468 1996 catalog_manager.cc:4928] TS 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): tablet af9984876006444aa460afd1604eb359 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]) successfully deleted
I20250905 07:50:20.106269 2733 raft_consensus.cc:383] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.110633 2733 raft_consensus.cc:738] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.110982 2599 tablet_service.cc:1468] Processing CreateTablet for tablet 84726ef69a6742a9b046a8833e406590 (DEFAULT_TABLE table=default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:20.110944 2733 consensus_queue.cc:260] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.111442 2599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84726ef69a6742a9b046a8833e406590. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:20.113345 2733 ts_tablet_manager.cc:1428] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250905 07:50:20.115451 2733 tablet_bootstrap.cc:492] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
@ 0x7f724fc0bb59 std::_Function_handler<>::_M_invoke() at ??:0
I20250905 07:50:20.120239 2012 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:20.120519 2012 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:20.121061 2733 tablet_bootstrap.cc:654] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.121920 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a57e0ff6b6a422aa0e69678b7bd6762" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:20.122156 2339 raft_consensus.cc:2466] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:20.122486 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e2858d296ec4022a337e6f3d6fa629d" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:20.122668 2340 raft_consensus.cc:2466] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:20.122934 2803 tablet_replica.cc:331] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:20.123194 2803 raft_consensus.cc:2241] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:20.123281 2416 leader_election.cc:304] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.123450 2803 raft_consensus.cc:2270] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:20.123768 2740 raft_consensus.cc:2802] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:20.123869 2740 raft_consensus.cc:491] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:20.123842 2416 leader_election.cc:304] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.124007 2740 raft_consensus.cc:3058] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.124089 2803 ts_tablet_manager.cc:1905] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:20.127066 2740 raft_consensus.cc:513] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:20.127882 2740 leader_election.cc:290] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Requested vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.128146 2740 raft_consensus.cc:2802] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:20.128261 2740 raft_consensus.cc:491] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:20.128367 2740 raft_consensus.cc:3058] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.128721 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a57e0ff6b6a422aa0e69678b7bd6762" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:20.128912 2340 raft_consensus.cc:3058] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.131022 2740 raft_consensus.cc:513] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.131655 2740 leader_election.cc:290] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Requested vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.132269 2340 raft_consensus.cc:2466] T 1a57e0ff6b6a422aa0e69678b7bd6762 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:20.132593 2803 ts_tablet_manager.cc:1918] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 07:50:20.132753 2803 log.cc:1199] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/af9984876006444aa460afd1604eb359
I20250905 07:50:20.133185 2803 ts_tablet_manager.cc:1939] T af9984876006444aa460afd1604eb359 P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:20.133287 2416 leader_election.cc:304] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.133606 2740 raft_consensus.cc:2802] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:20.133702 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e2858d296ec4022a337e6f3d6fa629d" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:20.133916 2340 raft_consensus.cc:3058] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.134349 1996 catalog_manager.cc:4928] TS 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): tablet af9984876006444aa460afd1604eb359 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]) successfully deleted
I20250905 07:50:20.134639 2804 tablet_replica.cc:331] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:20.134765 2740 raft_consensus.cc:695] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Running, Role: LEADER
I20250905 07:50:20.135231 2740 consensus_queue.cc:237] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:20.137867 2733 tablet_bootstrap.cc:492] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:20.138069 2733 ts_tablet_manager.cc:1397] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250905 07:50:20.139065 2733 raft_consensus.cc:357] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.139333 2733 raft_consensus.cc:383] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.139421 2733 raft_consensus.cc:738] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.137460 2340 raft_consensus.cc:2466] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:20.140643 2416 leader_election.cc:304] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.140956 2807 raft_consensus.cc:2802] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:20.141723 2621 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a57e0ff6b6a422aa0e69678b7bd6762" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:20.141976 2621 raft_consensus.cc:2466] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:20.142357 2620 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e2858d296ec4022a337e6f3d6fa629d" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:20.142604 2620 raft_consensus.cc:2466] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:20.142923 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a57e0ff6b6a422aa0e69678b7bd6762" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:20.143148 2619 raft_consensus.cc:3058] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.144129 2618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e2858d296ec4022a337e6f3d6fa629d" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:20.145192 2618 raft_consensus.cc:3058] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.139835 2804 raft_consensus.cc:2241] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:50:20.147096 2804 pending_rounds.cc:70] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Trying to abort 1 pending ops.
I20250905 07:50:20.147308 2804 pending_rounds.cc:77] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196912106517557248 op_type: NO_OP noop_request { }
I20250905 07:50:20.147559 2804 raft_consensus.cc:2887] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 07:50:20.147696 2804 raft_consensus.cc:2270] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:20.148743 2618 raft_consensus.cc:2466] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:20.148842 2804 ts_tablet_manager.cc:1905] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:20.148069 2733 consensus_queue.cc:260] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.151708 2619 raft_consensus.cc:2466] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:20.153651 2734 tablet_bootstrap.cc:492] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:20.153816 2734 ts_tablet_manager.cc:1397] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.014s
I20250905 07:50:20.154870 2734 raft_consensus.cc:357] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.155084 2734 raft_consensus.cc:383] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.155187 2734 raft_consensus.cc:738] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
@ 0xa774dd std::function<>::operator()() at ??:?
I20250905 07:50:20.155476 2734 consensus_queue.cc:260] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.159634 2734 ts_tablet_manager.cc:1428] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
@ 0x7f723c4f22b9 kudu::Thread::SuperviseThread() at ??:0
I20250905 07:50:20.163337 2734 tablet_bootstrap.cc:492] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:20.165485 2734 tablet_bootstrap.cc:654] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:20.165498 2735 tablet_bootstrap.cc:492] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:20.165674 2735 ts_tablet_manager.cc:1397] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.011s
I20250905 07:50:20.166417 2751 server_base.cc:1047] running on GCE node
I20250905 07:50:20.166631 2735 raft_consensus.cc:357] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.166832 2735 raft_consensus.cc:383] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.166953 2735 raft_consensus.cc:738] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.167194 2735 consensus_queue.cc:260] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.167714 2735 ts_tablet_manager.cc:1428] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 07:50:20.168427 2735 tablet_bootstrap.cc:492] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:20.168417 2733 ts_tablet_manager.cc:1428] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20250905 07:50:20.170153 2735 tablet_bootstrap.cc:654] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
W20250905 07:50:20.170720 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:20.172839 2751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
@ 0x7f723f9496db start_thread at ??:0
I20250905 07:50:20.175346 2807 raft_consensus.cc:695] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Running, Role: LEADER
I20250905 07:50:20.176133 2807 consensus_queue.cc:237] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.176458 2734 tablet_bootstrap.cc:492] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:20.176631 2734 ts_tablet_manager.cc:1397] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
@ 0x7f723863a71f clone at ??:0
I20250905 07:50:20.177523 2804 ts_tablet_manager.cc:1918] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:20.177699 2804 log.cc:1199] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/af9984876006444aa460afd1604eb359
I20250905 07:50:20.177783 2734 raft_consensus.cc:357] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.178045 2734 raft_consensus.cc:383] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.178128 2734 raft_consensus.cc:738] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.178323 2804 ts_tablet_manager.cc:1939] T af9984876006444aa460afd1604eb359 P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:20.178737 2734 consensus_queue.cc:260] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.181939 2734 ts_tablet_manager.cc:1428] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250905 07:50:20.183705 2751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:20.184180 2735 tablet_bootstrap.cc:492] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:20.184351 2735 ts_tablet_manager.cc:1397] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.001s
I20250905 07:50:20.184935 2751 hybrid_clock.cc:648] HybridClock initialized: now 1757058620184860 us; error 61 us; skew 500 ppm
I20250905 07:50:20.185118 2735 raft_consensus.cc:357] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.185307 2735 raft_consensus.cc:383] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:20.185375 2735 raft_consensus.cc:738] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.185424 2751 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:20.185668 2735 consensus_queue.cc:260] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.189687 2735 ts_tablet_manager.cc:1428] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250905 07:50:20.200073 2751 webserver.cc:480] Webserver started at http://127.1.178.188:42041/ using document root <none> and password file <none>
I20250905 07:50:20.201197 2751 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:20.201340 2751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:20.221873 2751 fs_manager.cc:714] Time spent opening directory manager: real 0.018s user 0.005s sys 0.009s
I20250905 07:50:20.221936 2807 raft_consensus.cc:491] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:20.222124 2807 raft_consensus.cc:513] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.222862 2807 leader_election.cc:290] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:20.223502 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a4ef48a5e67406297cf21413236f563" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:20.223764 2619 raft_consensus.cc:2466] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:20.224443 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a4ef48a5e67406297cf21413236f563" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:20.224570 2416 leader_election.cc:304] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:20.224634 2340 raft_consensus.cc:2466] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:20.224895 2807 raft_consensus.cc:2802] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:20.224994 2807 raft_consensus.cc:491] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:20.225111 2807 raft_consensus.cc:3058] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.227881 2807 raft_consensus.cc:513] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.228525 2807 leader_election.cc:290] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Requested vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:20.229045 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a4ef48a5e67406297cf21413236f563" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:20.229208 2340 raft_consensus.cc:3058] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.229152 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a4ef48a5e67406297cf21413236f563" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:20.229406 2619 raft_consensus.cc:3058] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.231611 2340 raft_consensus.cc:2466] T 7a4ef48a5e67406297cf21413236f563 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:20.232026 2416 leader_election.cc:304] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.232317 2807 raft_consensus.cc:2802] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:20.232460 2807 raft_consensus.cc:695] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Running, Role: LEADER
I20250905 07:50:20.232722 2807 consensus_queue.cc:237] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:20.232833 2619 raft_consensus.cc:2466] T 7a4ef48a5e67406297cf21413236f563 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:20.250449 2824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:20.253466 2751 fs_manager.cc:730] Time spent opening block manager: real 0.030s user 0.012s sys 0.016s
I20250905 07:50:20.253685 2751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
uuid: "81e2a8aedb144e61891de4dcd390ff35"
format_stamp: "Formatted at 2025-09-05 07:50:16 on dist-test-slave-tgv6"
I20250905 07:50:20.254786 2751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:20.279475 2751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:20.280640 2751 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:20.280953 2751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:20.304526 2751 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.188:39743
I20250905 07:50:20.304526 2879 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.188:39743 every 8 connection(s)
I20250905 07:50:20.306164 2751 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
I20250905 07:50:20.313409 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 2751
I20250905 07:50:20.315630 2880 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:20.320645 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Bootstrap starting.
I20250905 07:50:20.331857 2880 log.cc:826] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:20.341853 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 07:50:20.342403 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Bootstrap complete.
I20250905 07:50:20.347429 2880 raft_consensus.cc:357] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:20.348333 2880 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Initialized, Role: FOLLOWER
I20250905 07:50:20.349025 2880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:20.349900 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:20.350214 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:20.350908 2880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:20.379312 2898 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:20.379428 2898 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: loading cluster ID for follower catalog manager: success
I20250905 07:50:20.383294 2898 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: acquiring CA information for follower catalog manager: success
I20250905 07:50:20.385864 2898 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:20.403301 2750 raft_consensus.cc:491] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:20.403571 2750 raft_consensus.cc:513] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.404393 2750 leader_election.cc:290] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.404987 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28568a8557410c9104026b8f1cf3f1" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:20.405058 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28568a8557410c9104026b8f1cf3f1" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:20.405234 2483 raft_consensus.cc:2466] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:20.405290 2619 raft_consensus.cc:2466] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:20.405798 2273 leader_election.cc:304] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.406106 2750 raft_consensus.cc:2802] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:20.406217 2750 raft_consensus.cc:491] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:20.406322 2750 raft_consensus.cc:3058] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.409579 2750 raft_consensus.cc:513] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.410293 2750 leader_election.cc:290] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Requested vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.410878 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28568a8557410c9104026b8f1cf3f1" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:20.410904 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28568a8557410c9104026b8f1cf3f1" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:20.411094 2619 raft_consensus.cc:3058] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.411099 2483 raft_consensus.cc:3058] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.413936 2619 raft_consensus.cc:2466] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:20.414407 2273 leader_election.cc:304] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.414532 2483 raft_consensus.cc:2466] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:20.414788 2750 raft_consensus.cc:2802] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:20.415321 2750 raft_consensus.cc:695] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: Becoming Leader. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Running, Role: LEADER
I20250905 07:50:20.421941 2772 consensus_queue.cc:1035] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.416639 2750 consensus_queue.cc:237] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.424966 2385 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
W20250905 07:50:20.426921 2673 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:20.427711 2673 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.190:35357)
W20250905 07:50:20.427963 2673 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:20.428402 2673 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:20.429632 2673 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:20.437924 2906 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:50:20.430943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.178.190:35357 (local address 127.0.0.1:35904)
0905 07:50:20.437388 (+ 6445us) negotiation.cc:107] Waiting for socket to connect
0905 07:50:20.437578 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":6251,"threads_started":1}
W20250905 07:50:20.438640 2900 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:50:20.359175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.178.190:35357 (local address 127.0.0.1:35892)
0905 07:50:20.375205 (+ 16030us) negotiation.cc:107] Waiting for socket to connect
0905 07:50:20.375273 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0905 07:50:20.375577 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 07:50:20.437044 (+ 61467us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15607,"thread_start_us":19549,"threads_started":1}
W20250905 07:50:20.441792 2273 connection.cc:537] server connection from 127.0.0.1:52948 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 07:50:20.439042 2751 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250905 07:50:20.444550 2751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:20.467448 2902 raft_consensus.cc:491] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:20.467689 2902 raft_consensus.cc:513] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.468691 2902 leader_election.cc:290] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.469169 2845 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:39191; Asking this server to re-register.
I20250905 07:50:20.469659 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84726ef69a6742a9b046a8833e406590" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:20.469905 2483 raft_consensus.cc:2466] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:20.470212 2385 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:20.470639 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84726ef69a6742a9b046a8833e406590" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:20.470925 2619 raft_consensus.cc:2466] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:20.471467 2273 leader_election.cc:304] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.472290 2902 raft_consensus.cc:2802] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:20.472450 2902 raft_consensus.cc:491] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:20.472570 2902 raft_consensus.cc:3058] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.475579 2902 raft_consensus.cc:513] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.477572 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84726ef69a6742a9b046a8833e406590" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:20.477772 2483 raft_consensus.cc:3058] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.478680 2845 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:20.480593 2483 raft_consensus.cc:2466] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:20.481012 2273 leader_election.cc:304] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:20.481294 2902 leader_election.cc:290] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Requested vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.481513 2902 raft_consensus.cc:2802] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:20.481634 2902 raft_consensus.cc:695] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: Becoming Leader. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Running, Role: LEADER
I20250905 07:50:20.481935 2902 consensus_queue.cc:237] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:20.482354 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84726ef69a6742a9b046a8833e406590" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:20.482584 2619 raft_consensus.cc:3058] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:20.485879 2619 raft_consensus.cc:2466] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:20.493337 2770 consensus_queue.cc:1035] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.503433 2528 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:20.503652 2528 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
I20250905 07:50:20.513341 2770 consensus_queue.cc:1035] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.522377 2666 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:20.522609 2666 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
I20250905 07:50:20.525383 2844 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:45205; Asking this server to re-register.
I20250905 07:50:20.529016 2741 consensus_queue.cc:1035] T 3743b39493ed452e89b1e3223bb1ba88 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.530032 2666 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:20.531996 2845 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:20.540917 2772 consensus_queue.cc:1035] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.560427 2741 consensus_queue.cc:1035] T 09612f5ac871434399ea0cc0a7060c7b P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.560827 2845 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:38473; Asking this server to re-register.
I20250905 07:50:20.562520 2528 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:20.570436 2845 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:20.606266 2916 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:20.606434 2916 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:20.607148 2916 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:20.619295 2066 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:20.620550 2855 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
I20250905 07:50:20.621073 2855 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65033f3e4b44455389e29a5e1c9f82cd in term 3.
I20250905 07:50:20.621587 2068 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:20.621883 2916 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 07:50:20.621999 2916 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:20.622090 2916 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 4 FOLLOWER]: Advancing to term 5
I20250905 07:50:20.624050 2916 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:20.624682 2916 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 5 election: Requested vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:20.625077 2855 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35"
I20250905 07:50:20.625277 2855 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 3 FOLLOWER]: Advancing to term 5
W20250905 07:50:20.625948 2066 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:20.630049 2855 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65033f3e4b44455389e29a5e1c9f82cd in term 5.
I20250905 07:50:20.630472 2068 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:20.630837 2916 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 5 FOLLOWER]: Leader election won for term 5
I20250905 07:50:20.631333 2916 raft_consensus.cc:695] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 5 LEADER]: Becoming Leader. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Running, Role: LEADER
I20250905 07:50:20.631793 2916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:20.633240 2920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65033f3e4b44455389e29a5e1c9f82cd. Latest consensus state: current_term: 5 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:20.633493 2920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:20.634117 2921 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:20.636725 2921 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:20.637149 2921 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:20.637694 2921 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:20.638072 2921 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:20.638407 2921 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:20.638751 2921 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:20.639134 2921 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:20.639493 2921 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:20.639824 2921 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:20.641615 2921 tablet_loader.cc:96] loaded metadata for tablet 083c74057a6945288b259678e0ff2223 (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:20.641973 2921 tablet_loader.cc:96] loaded metadata for tablet 09612f5ac871434399ea0cc0a7060c7b (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:20.642454 2921 tablet_loader.cc:96] loaded metadata for tablet 0c28568a8557410c9104026b8f1cf3f1 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:20.642732 2921 tablet_loader.cc:96] loaded metadata for tablet 1a57e0ff6b6a422aa0e69678b7bd6762 (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:20.643018 2921 tablet_loader.cc:96] loaded metadata for tablet 3743b39493ed452e89b1e3223bb1ba88 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:20.643289 2921 tablet_loader.cc:96] loaded metadata for tablet 7a4ef48a5e67406297cf21413236f563 (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:20.643571 2921 tablet_loader.cc:96] loaded metadata for tablet 84726ef69a6742a9b046a8833e406590 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:20.643838 2921 tablet_loader.cc:96] loaded metadata for tablet 8b52a829672943da8121d121052697ab (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:20.644181 2921 tablet_loader.cc:96] loaded metadata for tablet 9e2858d296ec4022a337e6f3d6fa629d (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:20.644750 2921 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:20.645677 2921 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:20.645758 2921 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:20.646961 2921 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:20.647864 2921 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Loaded TSK: 0
I20250905 07:50:20.648830 2921 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:20.649744 2921 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:20.650874 2921 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:20.656250 2082 master_service.cc:948] ReplaceTablet: received request to replace tablet 3743b39493ed452e89b1e3223bb1ba88 from {username='slave'} at 127.0.0.1:42612
I20250905 07:50:20.660310 2073 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a57e0ff6b6a422aa0e69678b7bd6762 from {username='slave'} at 127.0.0.1:42612
I20250905 07:50:20.658937 2081 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42612:
name: "default.Table_c6a86ab58c8f4eddbf3e0c4edf058247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:20.657647 2074 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42612:
name: "default.Table_347a8cac73e54038b34319425f8f3bd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:20.660755 2076 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42612:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:20.663151 2076 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42612:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:20.660454 2077 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42612:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:20.663537 2077 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42612:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:20.661113 2075 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42612:
name: "default.Table_4907c91834874a1a88a3312b6a0be96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:20.659610 2079 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42612:
name: "default.Table_b3cf8c80578f4b4f84327aedafee6a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:20.659271 2080 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42612:
name: "default.Table_cd8e6f84c6614cd58e55c59c90383205"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:20.660120 2078 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42612:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
I20250905 07:50:20.665131 2078 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42612:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
W20250905 07:50:20.669871 2066 consensus_peers.cc:489] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd -> Peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Couldn't send request to peer 9dbf15725a7342da830f57c0535eb72d. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250905 07:50:20.670455 2254 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250905 07:50:20.672376 2855 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Refusing update from remote peer 65033f3e4b44455389e29a5e1c9f82cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
@ 0x7f61c47d0dcd google::LogMessage::Fail() at ??:0
I20250905 07:50:20.673534 2920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 07:50:20.673755 2920 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [LEADER]: Peer 81e2a8aedb144e61891de4dcd390ff35 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
@ 0x7f61c47d4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f61c47d07cc google::LogMessage::Flush() at ??:0
I20250905 07:50:20.676419 2855 pending_rounds.cc:85] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Aborting all ops after (but not including) 4
I20250905 07:50:20.676551 2855 pending_rounds.cc:107] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250905 07:50:20.676715 2855 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7f61c47d0a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f61dac4c7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f61dac4b061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f61dac52cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f61dac52b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f61c75392b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61ca9906db start_thread at ??:0
@ 0x7f61c368171f clone at ??:0
I20250905 07:50:20.717419 2926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:20.717772 2926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:20.718190 2928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:20.718549 2928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:20.718897 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65033f3e4b44455389e29a5e1c9f82cd. Latest consensus state: current_term: 5 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:20.719103 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:20.733736 2924 mvcc.cc:204] Tried to move back new op lower bound from 7196912107908526080 to 7196912098533634048. Current Snapshot: MvccSnapshot[applied={T|T < 7196912107908526080 or (T in {7196912107908526080})}]
I20250905 07:50:20.772357 2772 consensus_queue.cc:1035] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.782284 2772 consensus_queue.cc:1035] T 083c74057a6945288b259678e0ff2223 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 07:50:20.892294 2552 connection.cc:537] client connection to 127.1.178.189:43391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:20.892333 2416 connection.cc:537] client connection to 127.1.178.189:43391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:20.892371 2273 connection.cc:537] client connection to 127.1.178.189:43391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:20.894586 2831 connection.cc:537] server connection from 127.0.0.1:45914 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 07:50:20.947499 2807 consensus_queue.cc:1035] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.954445 2945 consensus_queue.cc:1035] T 1a57e0ff6b6a422aa0e69678b7bd6762 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:20.997435 2946 consensus_queue.cc:1035] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.002555 2946 consensus_queue.cc:1035] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.177673 2884 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:21.177901 2884 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:21.179605 2884 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:21.180578 2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:21.182051 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:21.182278 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:21.182379 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:21.182723 2884 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 07:50:21.207805 2902 consensus_queue.cc:1035] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.215507 2953 consensus_queue.cc:1035] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.458842 2946 consensus_queue.cc:1035] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.465314 2807 consensus_queue.cc:1035] T 7a4ef48a5e67406297cf21413236f563 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.628723 2954 consensus_queue.cc:1035] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.634516 2953 consensus_queue.cc:1035] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:21.766063 2965 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:21.766243 2965 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:21.766916 2965 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:21.768904 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:21.769063 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:21.769145 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:21.769423 2965 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 07:50:22.495620 2976 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:22.495844 2976 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:22.496567 2976 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:22.499429 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:22.499607 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:22.499701 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:22.499994 2976 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 07:50:23.282518 2990 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:23.282744 2990 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:23.283371 2990 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:23.287824 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:23.288094 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:23.288228 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:23.288489 2990 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 07:50:23.630882 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--webserver_interface=127.1.178.190
--webserver_port=34789
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:23.960263 2998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:23.960582 2998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:23.960714 2998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:23.969009 2998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:23.969099 2998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:23.969172 2998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:23.969240 2998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:23.979231 2998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.178.190
--webserver_port=34789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:23.980891 2998 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:23.982592 2998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:23.990020 3005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:23.990025 3006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:23.990514 3008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:23.991281 2998 server_base.cc:1047] running on GCE node
I20250905 07:50:23.992054 2998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:23.993651 2998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:23.994879 2998 hybrid_clock.cc:648] HybridClock initialized: now 1757058623994816 us; error 51 us; skew 500 ppm
I20250905 07:50:23.995222 2998 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:23.997251 2998 webserver.cc:480] Webserver started at http://127.1.178.190:34789/ using document root <none> and password file <none>
I20250905 07:50:23.997923 2998 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:23.998027 2998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:24.003122 2998 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20250905 07:50:24.006685 3017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:24.008014 2998 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250905 07:50:24.008170 2998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
uuid: "9dbf15725a7342da830f57c0535eb72d"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:24.008855 2998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:24.034564 2998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:24.035526 2998 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:24.035805 2998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:24.057013 2998 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.190:35357
I20250905 07:50:24.057031 3069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.190:35357 every 8 connection(s)
I20250905 07:50:24.058785 2998 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
I20250905 07:50:24.059402 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 2998
I20250905 07:50:24.067245 3070 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:24.070817 3070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap starting.
I20250905 07:50:24.082029 3070 log.cc:826] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:24.108911 3070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:50:24.110841 3070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap complete.
I20250905 07:50:24.118534 3070 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:24.119220 3070 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Initialized, Role: FOLLOWER
I20250905 07:50:24.119804 3070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:24.120545 3074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:24.120779 3074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:24.121551 3070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:24.137871 3088 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:24.138068 3088 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: loading cluster ID for follower catalog manager: success
I20250905 07:50:24.142978 3088 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: acquiring CA information for follower catalog manager: success
I20250905 07:50:24.146283 3088 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:24.146580 2998 master_runner.cc:186] Error getting master registration for 127.1.178.189:43391: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111),
I20250905 07:50:24.150998 2998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:24.198076 2529 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:24.201583 3035 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:41403; Asking this server to re-register.
I20250905 07:50:24.202291 2529 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:24.203816 3035 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:24.512030 2667 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:24.513882 3035 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:54799; Asking this server to re-register.
I20250905 07:50:24.514506 2667 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:24.515640 3035 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:24.516757 3074 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:24.516945 3074 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:24.518661 3074 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:24.519224 3018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
W20250905 07:50:24.521044 3018 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:24.525051 2855 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
I20250905 07:50:24.525310 2855 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dbf15725a7342da830f57c0535eb72d in current term 5: Already voted for candidate 65033f3e4b44455389e29a5e1c9f82cd in this term.
I20250905 07:50:24.525856 3020 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:24.526196 3074 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 4 FOLLOWER]: Advancing to term 5
I20250905 07:50:24.530426 3074 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 07:50:24.552284 2386 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:24.554105 3035 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:43319; Asking this server to re-register.
I20250905 07:50:24.554813 2386 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:24.556280 3035 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:25.173488 3099 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:25.173694 3099 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:25.174369 3099 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:25.174918 2855 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 6 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
I20250905 07:50:25.175196 2855 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dbf15725a7342da830f57c0535eb72d for term 6 because replica has last-logged OpId of term: 5 index: 36, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 33.
W20250905 07:50:25.176440 3018 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:25.176548 3018 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:25.176820 3099 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 07:50:25.282233 3103 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:25.282461 3103 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:25.283097 3103 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:25.285835 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:25.289543 3045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 6 candidate_status { last_received { term: 5 index: 36 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d" is_pre_election: true
I20250905 07:50:25.290025 3045 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 5.
I20250905 07:50:25.290637 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd
I20250905 07:50:25.290949 3103 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 07:50:25.291071 3103 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:25.291182 3103 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 5 FOLLOWER]: Advancing to term 6
I20250905 07:50:25.293097 3103 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:25.293589 3103 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 election: Requested vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
I20250905 07:50:25.294046 3045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 6 candidate_status { last_received { term: 5 index: 36 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d"
I20250905 07:50:25.294209 3045 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 5 FOLLOWER]: Advancing to term 6
W20250905 07:50:25.294905 2829 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:25.296240 3045 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 6.
I20250905 07:50:25.296625 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd
I20250905 07:50:25.296917 3103 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 07:50:25.297258 3103 raft_consensus.cc:695] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 6 LEADER]: Becoming Leader. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Running, Role: LEADER
I20250905 07:50:25.297669 3103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.36, Last appended by leader: 36, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:25.299137 3108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 6 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:25.299353 3108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:25.448451 2751 master_runner.cc:186] Error getting master registration for 127.1.178.189:43391: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111),
I20250905 07:50:25.448642 2751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:25.803156 3115 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:25.803336 3115 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:25.804082 3115 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:25.804626 2855 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 7 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
W20250905 07:50:25.805797 3018 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:25.805898 3018 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:25.806169 3115 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 07:50:25.905611 3045 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Refusing update from remote peer 81e2a8aedb144e61891de4dcd390ff35: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 6 index: 37. (index mismatch)
I20250905 07:50:25.906392 3118 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 33, Time since last communication: 0.000s
I20250905 07:50:25.911408 3115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 6 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:25.911612 3115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:25.915390 3118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:25.915740 3118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:25.916127 3118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:25.916388 3118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:25.919626 3115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:25.919579 3130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:25.919853 3115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:25.919881 3130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
W20250905 07:50:25.920120 2829 consensus_peers.cc:489] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 -> Peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Couldn't send request to peer 65033f3e4b44455389e29a5e1c9f82cd. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:25.944833 3110 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:25.947502 3110 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:25.948029 3110 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:25.948398 3110 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:25.948776 3110 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:25.949127 3110 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:25.949478 3110 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:25.949833 3110 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:25.950176 3110 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:25.950491 3110 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:25.952375 3110 tablet_loader.cc:96] loaded metadata for tablet 083c74057a6945288b259678e0ff2223 (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:25.952761 3110 tablet_loader.cc:96] loaded metadata for tablet 09612f5ac871434399ea0cc0a7060c7b (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:25.953472 3110 tablet_loader.cc:96] loaded metadata for tablet 0c28568a8557410c9104026b8f1cf3f1 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:25.954129 3110 tablet_loader.cc:96] loaded metadata for tablet 7a4ef48a5e67406297cf21413236f563 (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:25.954480 3110 tablet_loader.cc:96] loaded metadata for tablet 84726ef69a6742a9b046a8833e406590 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:25.954839 3110 tablet_loader.cc:96] loaded metadata for tablet 8b52a829672943da8121d121052697ab (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:25.955188 3110 tablet_loader.cc:96] loaded metadata for tablet 95be079f58414b529a5f2009f56bb009 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:25.955550 3110 tablet_loader.cc:96] loaded metadata for tablet 9e2858d296ec4022a337e6f3d6fa629d (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:25.956157 3110 tablet_loader.cc:96] loaded metadata for tablet c022e7ee959d4d19958437a1ed855bbd (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:25.956714 3110 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:25.958104 3110 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:25.958197 3110 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:25.959851 3110 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:25.961136 3110 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Loaded TSK: 0
I20250905 07:50:25.962209 3110 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:25.963375 3110 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:25.964532 3110 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:26.031376 2844 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45900:
name: "default.Table_06530af7099e45c7b150408f8c1a3436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:26.031890 2840 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_df9d719332a94674bf8d0b4f1f9d6a7f" } new_table_name: "default.Table_30a6668a12364eedb06bd2b9bfbf4b31" modify_external_catalogs: true
I20250905 07:50:26.031430 2845 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45900:
name: "default.Table_6abbacd07c2f4ab4b74761cd100dcf58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:26.032778 2842 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_146568e744294fdd8f082cdbd8451039" } new_table_name: "default.Table_bcc153425f724fa0aee1ba610df560ce" modify_external_catalogs: true
I20250905 07:50:26.033610 2836 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
I20250905 07:50:26.036343 2838 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_fbd6c4d89fd54fdd9765576cc3cf5cbc" } new_table_name: "default.Table_68b9ab7fcb3245ad84e0fe0e5c05d746" modify_external_catalogs: true
I20250905 07:50:26.036093 2839 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45900:
name: "default.Table_8f62349a373e45b8ae063dafbfc3cdcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:26.036391 2843 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_a908ec5ae1aa46e8b872ff681d610476" } new_table_name: "default.Table_23be441c886b4e17929155f056fdb23b" modify_external_catalogs: true
I20250905 07:50:26.033990 2836 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
I20250905 07:50:26.033861 2837 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:26.037392 2841 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_9a0733038d2145b2a3ea38308d4b6fa8" } new_table_name: "default.Table_96eada578e0d465cb55ab0848c68acaa" modify_external_catalogs: true
I20250905 07:50:26.037493 2837 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
F20250905 07:50:26.038951 2897 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7feec73a3dcd google::LogMessage::Fail() at ??:0
@ 0x7feec73a7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7feec73a37cc google::LogMessage::Flush() at ??:0
@ 0x7feec73a3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7feedd81f7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7feedd81e061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7feedd825cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7feedd825b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7feeca10c2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feecd5636db start_thread at ??:0
@ 0x7feec625471f clone at ??:0
W20250905 07:50:26.241552 2418 connection.cc:537] client connection to 127.1.178.188:39743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:26.242329 2675 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.188:39743)
W20250905 07:50:26.242632 2675 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.188:39743)
W20250905 07:50:26.242913 2675 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.188:39743)
W20250905 07:50:26.243474 2675 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.188:39743)
I20250905 07:50:26.414229 3130 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:26.414456 3130 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:26.415212 3130 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:26.417297 3018 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
W20250905 07:50:26.417438 3020 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:26.417589 3020 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:26.417845 3130 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 07:50:26.514267 2666 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
I20250905 07:50:26.887326 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--webserver_interface=127.1.178.189
--webserver_port=37293
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 07:50:26.934676 3154 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:26.934832 3154 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:26.935453 3154 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:26.937484 3018 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
W20250905 07:50:26.937633 3020 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:26.937753 3020 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:26.938023 3154 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 07:50:27.227741 3153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:27.228108 3153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:27.228200 3153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:27.236457 3153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:27.236550 3153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:27.236603 3153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:27.236650 3153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:27.246975 3153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.178.189
--webserver_port=37293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:27.248627 3153 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:27.250483 3153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:27.258963 3162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:27.259200 3163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:27.260200 3165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:27.261209 3153 server_base.cc:1047] running on GCE node
I20250905 07:50:27.261957 3153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:27.263298 3153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:27.264508 3153 hybrid_clock.cc:648] HybridClock initialized: now 1757058627264454 us; error 39 us; skew 500 ppm
I20250905 07:50:27.264878 3153 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:27.267038 3153 webserver.cc:480] Webserver started at http://127.1.178.189:37293/ using document root <none> and password file <none>
I20250905 07:50:27.267786 3153 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:27.267910 3153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:27.273208 3153 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 07:50:27.276149 3171 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:27.277520 3153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:50:27.277693 3153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
uuid: "65033f3e4b44455389e29a5e1c9f82cd"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:27.278404 3153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:27.297494 3153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:27.298367 3153 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:27.298664 3153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:27.318665 3153 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.189:43391
I20250905 07:50:27.318684 3222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.189:43391 every 8 connection(s)
I20250905 07:50:27.320387 3153 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
I20250905 07:50:27.320859 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 3153
I20250905 07:50:27.329447 3223 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:27.333608 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap starting.
I20250905 07:50:27.344261 3223 log.cc:826] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:27.373322 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250905 07:50:27.373926 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap complete.
I20250905 07:50:27.378979 3223 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 5 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:27.381757 3223 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Initialized, Role: FOLLOWER
I20250905 07:50:27.382407 3223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 5.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:27.383113 3228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:27.383368 3228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:27.384192 3223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:27.395834 3242 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:27.396046 3242 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: loading cluster ID for follower catalog manager: success
I20250905 07:50:27.402438 3242 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: acquiring CA information for follower catalog manager: success
I20250905 07:50:27.405267 3242 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:27.432333 3153 master_runner.cc:186] Error getting master registration for 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111),
I20250905 07:50:27.436957 3153 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:27.519745 2668 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:27.523092 3188 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:55507; Asking this server to re-register.
I20250905 07:50:27.523829 2668 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:27.525439 3188 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:27.562299 2387 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:27.564553 3188 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:47271; Asking this server to re-register.
I20250905 07:50:27.565311 2387 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:27.566836 3188 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:27.705423 2530 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:27.707531 3188 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:46319; Asking this server to re-register.
I20250905 07:50:27.708222 2530 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:27.709775 3188 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:27.785140 3248 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:27.785379 3248 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:27.786041 3248 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:27.788439 3020 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:27.792634 3198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 7 candidate_status { last_received { term: 6 index: 37 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:27.793133 3198 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 5.
I20250905 07:50:27.793578 3018 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:27.793877 3248 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 07:50:27.793982 3248 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:27.794078 3248 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 6 FOLLOWER]: Advancing to term 7
I20250905 07:50:27.796245 3248 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:27.796875 3248 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 election: Requested vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:27.797463 3198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 7 candidate_status { last_received { term: 6 index: 37 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:27.797701 3198 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 5 FOLLOWER]: Advancing to term 7
W20250905 07:50:27.798337 3020 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:27.801746 3198 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 7.
I20250905 07:50:27.802279 3018 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:27.802642 3248 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 7 FOLLOWER]: Leader election won for term 7
I20250905 07:50:27.803068 3248 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 7 LEADER]: Becoming Leader. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Running, Role: LEADER
I20250905 07:50:27.803539 3248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 6.37, Last appended by leader: 37, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:27.805292 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 7 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:27.805626 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:27.806342 3254 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:27.808974 3254 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:27.809443 3254 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:27.809836 3254 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:27.810190 3254 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:27.810526 3254 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:27.810909 3254 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:27.811249 3254 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:27.811612 3254 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:27.812007 3254 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:27.813803 3254 tablet_loader.cc:96] loaded metadata for tablet 083c74057a6945288b259678e0ff2223 (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:27.814215 3254 tablet_loader.cc:96] loaded metadata for tablet 09612f5ac871434399ea0cc0a7060c7b (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:27.814757 3254 tablet_loader.cc:96] loaded metadata for tablet 0c28568a8557410c9104026b8f1cf3f1 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:27.815366 3254 tablet_loader.cc:96] loaded metadata for tablet 7a4ef48a5e67406297cf21413236f563 (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:27.815706 3254 tablet_loader.cc:96] loaded metadata for tablet 84726ef69a6742a9b046a8833e406590 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:27.816078 3254 tablet_loader.cc:96] loaded metadata for tablet 8b52a829672943da8121d121052697ab (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:27.816360 3254 tablet_loader.cc:96] loaded metadata for tablet 95be079f58414b529a5f2009f56bb009 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:27.816668 3254 tablet_loader.cc:96] loaded metadata for tablet 9e2858d296ec4022a337e6f3d6fa629d (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:27.817114 3254 tablet_loader.cc:96] loaded metadata for tablet c022e7ee959d4d19958437a1ed855bbd (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:27.817553 3254 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:27.818617 3254 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:27.818703 3254 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:27.820060 3254 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:27.821040 3254 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Loaded TSK: 0
I20250905 07:50:27.821944 3254 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:27.822959 3254 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:27.823856 3254 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:27.834746 3035 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45952:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:27.834682 3033 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45952:
name: "default.Table_7c0359e055804d0b9a84bd4dd5e60634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:27.835026 3035 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45952:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:27.835345 3029 master_service.cc:948] ReplaceTablet: received request to replace tablet 09612f5ac871434399ea0cc0a7060c7b from {username='slave'} at 127.0.0.1:45952
I20250905 07:50:27.835757 3030 master_service.cc:948] ReplaceTablet: received request to replace tablet 083c74057a6945288b259678e0ff2223 from {username='slave'} at 127.0.0.1:45952
I20250905 07:50:27.836113 3028 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a57e0ff6b6a422aa0e69678b7bd6762 from {username='slave'} at 127.0.0.1:45952
I20250905 07:50:27.836251 3031 master_service.cc:948] ReplaceTablet: received request to replace tablet af9984876006444aa460afd1604eb359 from {username='slave'} at 127.0.0.1:45952
I20250905 07:50:27.837186 3031 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45952:
table { table_name: "default.Table_a908ec5ae1aa46e8b872ff681d610476" } new_table_name: "default.Table_23be441c886b4e17929155f056fdb23b" modify_external_catalogs: true
I20250905 07:50:27.837359 3027 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45952:
name: "default.Table_347a8cac73e54038b34319425f8f3bd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:27.837875 3032 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45952:
name: "default.Table_e3daa55fc9e5447f82536064de82c58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:27.838416 3026 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45952:
table { table_name: "default.Table_df9d719332a94674bf8d0b4f1f9d6a7f" } new_table_name: "default.Table_30a6668a12364eedb06bd2b9bfbf4b31" modify_external_catalogs: true
I20250905 07:50:27.838758 3028 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45952:
table { table_name: "default.Table_146568e744294fdd8f082cdbd8451039" } new_table_name: "default.Table_bcc153425f724fa0aee1ba610df560ce" modify_external_catalogs: true
I20250905 07:50:27.840188 3025 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45952:
table { table_name: "default.Table_9a0733038d2145b2a3ea38308d4b6fa8" } new_table_name: "default.Table_96eada578e0d465cb55ab0848c68acaa" modify_external_catalogs: true
W20250905 07:50:27.844599 3020 consensus_peers.cc:489] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d -> Peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Couldn't send request to peer 81e2a8aedb144e61891de4dcd390ff35. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:27.844956 3198 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Refusing update from remote peer 9dbf15725a7342da830f57c0535eb72d: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 7 index: 39. (index mismatch)
F20250905 07:50:27.845465 3087 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250905 07:50:27.845824 3252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 33, Time since last communication: 0.000s
@ 0x7f7363474dcd google::LogMessage::Fail() at ??:0
@ 0x7f7363478b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f73634747cc google::LogMessage::Flush() at ??:0
@ 0x7f7363474a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f73798f07b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250905 07:50:27.867111 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:27.867529 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:27.868175 3228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 7 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:27.868408 3228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:27.869041 3258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:27.869370 3258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f73798ef061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250905 07:50:27.870242 3258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:27.870406 3258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:27.874787 3029 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet 09612f5ac871434399ea0cc0a7060c7b
I20250905 07:50:27.874866 3030 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet 083c74057a6945288b259678e0ff2223
@ 0x7f73798f6cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250905 07:50:27.876574 3029 catalog_manager.cc:6652] ReplaceTablet: tablet 09612f5ac871434399ea0cc0a7060c7b deleted and replaced by tablet f7fceab9655d4d9ea0e142f0e3a4d06e
@ 0x7f73798f6b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250905 07:50:27.881532 3029 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45952:
name: "default.Table_6abbacd07c2f4ab4b74761cd100dcf58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:27.881784 3030 catalog_manager.cc:6652] ReplaceTablet: tablet 083c74057a6945288b259678e0ff2223 deleted and replaced by tablet ff5338af10954f3f9ee797c37a1ca9cc
I20250905 07:50:27.882272 3030 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45952:
name: "default.Table_06530af7099e45c7b150408f8c1a3436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:27.882731 3248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:27.883025 3248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f73661dd2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73696346db start_thread at ??:0
@ 0x7f736232571f clone at ??:0
W20250905 07:50:28.061865 2673 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:28.062193 3175 connection.cc:537] server connection from 127.0.0.1:46924 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:28.062335 3270 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:50:27.877307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.1.178.131:45275)
0905 07:50:27.896695 (+ 19388us) server_negotiation.cc:207] Beginning negotiation
0905 07:50:27.896704 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0905 07:50:27.896730 (+ 26us) server_negotiation.cc:408] Connection header received
0905 07:50:27.896814 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:50:27.896820 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:50:27.896907 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:50:27.897022 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:50:28.061286 (+164264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33324: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":19227,"thread_start_us":19111,"threads_started":1}
W20250905 07:50:28.062579 2673 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.190:35357)
W20250905 07:50:28.062673 2552 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:28.062891 2673 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:28.063097 2673 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:28.063638 3271 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:50:27.877474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.1.178.129:45773)
0905 07:50:27.895367 (+ 17893us) server_negotiation.cc:207] Beginning negotiation
0905 07:50:27.895376 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0905 07:50:27.895410 (+ 34us) server_negotiation.cc:408] Connection header received
0905 07:50:27.895514 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:50:27.895524 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:50:27.895644 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:50:27.895791 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:50:28.061324 (+165533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35526: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":17695,"thread_start_us":17539,"threads_started":1}
W20250905 07:50:28.064118 2673 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:28.064872 3272 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 07:50:27.879027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.1.178.130:37221)
0905 07:50:27.896440 (+ 17413us) server_negotiation.cc:207] Beginning negotiation
0905 07:50:27.896448 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0905 07:50:27.896468 (+ 20us) server_negotiation.cc:408] Connection header received
0905 07:50:27.896542 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0905 07:50:27.896550 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0905 07:50:27.896629 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0905 07:50:27.896731 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0905 07:50:28.061325 (+164594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43688: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":17243,"thread_start_us":17098,"threads_started":1}
I20250905 07:50:28.393260 3281 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:28.393463 3281 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:28.395292 3281 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:28.396342 3174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:28.397640 3172 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:28.397841 3174 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:28.398046 3174 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:28.398388 3281 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 07:50:29.122879 3286 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:29.123116 3286 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:29.123798 3286 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:29.125957 3174 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:29.126129 3172 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:29.126250 3172 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:29.126554 3286 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 07:50:30.079638 3291 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:30.079842 3291 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:30.080680 3291 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:30.082551 3174 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:30.082698 3172 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:30.082821 3172 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:30.083024 3291 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 07:50:30.325748 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.188:39743
--webserver_interface=127.1.178.188
--webserver_port=42041
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:30.676951 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:30.677300 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:30.677397 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:30.687980 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:30.688123 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:30.688220 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:30.688301 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:30.698845 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.188:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.178.188
--webserver_port=42041
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:30.700625 3295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:30.702394 3295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:30.710534 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:30.710490 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:30.711124 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:30.711746 3295 server_base.cc:1047] running on GCE node
I20250905 07:50:30.712479 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:30.713836 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:30.715057 3295 hybrid_clock.cc:648] HybridClock initialized: now 1757058630715004 us; error 45 us; skew 500 ppm
I20250905 07:50:30.715396 3295 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:30.717619 3295 webserver.cc:480] Webserver started at http://127.1.178.188:42041/ using document root <none> and password file <none>
I20250905 07:50:30.718354 3295 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:30.718461 3295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:30.723840 3295 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20250905 07:50:30.727190 3310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:30.728571 3295 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250905 07:50:30.728744 3295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
uuid: "81e2a8aedb144e61891de4dcd390ff35"
format_stamp: "Formatted at 2025-09-05 07:50:16 on dist-test-slave-tgv6"
I20250905 07:50:30.729460 3295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:30.742120 3295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:30.742997 3295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:30.743280 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:30.764683 3295 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.188:39743
I20250905 07:50:30.764685 3361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.188:39743 every 8 connection(s)
I20250905 07:50:30.766544 3295 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-2/data/info.pb
I20250905 07:50:30.775597 3362 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:30.775945 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 3295
I20250905 07:50:30.781569 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Bootstrap starting.
I20250905 07:50:30.794705 3362 log.cc:826] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:30.821260 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=1 applied=37 ignored=0} inserts{seen=26 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:50:30.821930 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Bootstrap complete.
I20250905 07:50:30.827548 3362 raft_consensus.cc:357] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:30.828238 3362 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Initialized, Role: FOLLOWER
I20250905 07:50:30.828919 3362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 6.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:30.829789 3367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:30.830042 3367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:30.830792 3362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:30.846091 3381 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:30.846210 3381 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: loading cluster ID for follower catalog manager: success
I20250905 07:50:30.840967 3295 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:50:30.847481 3295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:30.852273 3381 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: acquiring CA information for follower catalog manager: success
I20250905 07:50:30.855382 3381 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:31.034514 2666 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:31.034787 2666 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
I20250905 07:50:31.038792 3327 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:52957; Asking this server to re-register.
I20250905 07:50:31.039700 2666 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:31.041765 3327 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:31.070189 3385 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:31.070420 3385 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:31.071115 3385 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:31.073827 3172 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:31.079154 3337 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 41 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35" is_pre_election: true
I20250905 07:50:31.079805 3337 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65033f3e4b44455389e29a5e1c9f82cd in term 6.
I20250905 07:50:31.080441 3174 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:31.080821 3385 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 07:50:31.080952 3385 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:31.081065 3385 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 7 FOLLOWER]: Advancing to term 8
I20250905 07:50:31.083459 3385 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:31.084322 3385 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 election: Requested vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:31.085157 3337 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 41 } } ignore_live_leader: false dest_uuid: "81e2a8aedb144e61891de4dcd390ff35"
I20250905 07:50:31.085405 3337 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 6 FOLLOWER]: Advancing to term 8
W20250905 07:50:31.086480 3172 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:31.091534 3337 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65033f3e4b44455389e29a5e1c9f82cd in term 8.
I20250905 07:50:31.093600 3174 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:31.093977 3385 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 8 FOLLOWER]: Leader election won for term 8
I20250905 07:50:31.094496 3385 raft_consensus.cc:695] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 8 LEADER]: Becoming Leader. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Running, Role: LEADER
I20250905 07:50:31.094991 3385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 7.41, Last appended by leader: 41, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:31.096644 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65033f3e4b44455389e29a5e1c9f82cd. Latest consensus state: current_term: 8 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:31.096946 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:31.261143 2528 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:31.261128 2385 heartbeater.cc:344] Connected to a master server at 127.1.178.188:39743
I20250905 07:50:31.262856 3327 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:49595; Asking this server to re-register.
I20250905 07:50:31.262862 3326 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:59515; Asking this server to re-register.
I20250905 07:50:31.263571 2528 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:31.263577 2385 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:31.265106 3326 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:31.265182 3327 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:31.568079 3337 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Refusing update from remote peer 65033f3e4b44455389e29a5e1c9f82cd: Log matching property violated. Preceding OpId in replica: term: 6 index: 37. Preceding OpId from leader: term: 8 index: 42. (index mismatch)
I20250905 07:50:31.568977 3390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 37, Time since last communication: 0.000s
I20250905 07:50:31.577505 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65033f3e4b44455389e29a5e1c9f82cd. Latest consensus state: current_term: 8 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:31.577873 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:31.577994 3395 mvcc.cc:204] Tried to move back new op lower bound from 7196912139647201280 to 7196912139486949376. Current Snapshot: MvccSnapshot[applied={T|T < 7196912139647201280 or (T in {7196912139647201280})}]
I20250905 07:50:31.580090 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:31.580327 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:31.580451 3397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:31.580785 3397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:31.581784 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:31.581980 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: LEADER
W20250905 07:50:31.582739 3172 consensus_peers.cc:489] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd -> Peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Couldn't send request to peer 9dbf15725a7342da830f57c0535eb72d. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:31.614593 3392 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:31.617318 3392 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:31.617775 3392 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:31.618089 3392 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:31.618386 3392 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:31.618667 3392 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:31.618974 3392 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:31.619254 3392 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:31.619542 3392 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:31.619818 3392 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:31.622210 3392 tablet_loader.cc:96] loaded metadata for tablet 0c28568a8557410c9104026b8f1cf3f1 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:31.622807 3392 tablet_loader.cc:96] loaded metadata for tablet 7a4ef48a5e67406297cf21413236f563 (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:31.623157 3392 tablet_loader.cc:96] loaded metadata for tablet 84726ef69a6742a9b046a8833e406590 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:31.623454 3392 tablet_loader.cc:96] loaded metadata for tablet 8b52a829672943da8121d121052697ab (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:31.623718 3392 tablet_loader.cc:96] loaded metadata for tablet 95be079f58414b529a5f2009f56bb009 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:31.623996 3392 tablet_loader.cc:96] loaded metadata for tablet 9e2858d296ec4022a337e6f3d6fa629d (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:31.624400 3392 tablet_loader.cc:96] loaded metadata for tablet c022e7ee959d4d19958437a1ed855bbd (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:31.624785 3392 tablet_loader.cc:96] loaded metadata for tablet f7fceab9655d4d9ea0e142f0e3a4d06e (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:31.625037 3392 tablet_loader.cc:96] loaded metadata for tablet ff5338af10954f3f9ee797c37a1ca9cc (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:31.625265 3392 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:31.626269 3392 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:31.626356 3392 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:31.627563 3392 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:31.628597 3392 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Loaded TSK: 0
I20250905 07:50:31.629479 3392 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:31.630347 3392 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:31.631167 3392 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:31.668264 3188 master_service.cc:948] ReplaceTablet: received request to replace tablet 9e2858d296ec4022a337e6f3d6fa629d from {username='slave'} at 127.0.0.1:46920
I20250905 07:50:31.668282 3187 master_service.cc:948] ReplaceTablet: received request to replace tablet 8b52a829672943da8121d121052697ab from {username='slave'} at 127.0.0.1:46920
I20250905 07:50:31.669092 3185 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:31.669443 3185 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:31.669668 3183 master_service.cc:948] ReplaceTablet: received request to replace tablet 3743b39493ed452e89b1e3223bb1ba88 from {username='slave'} at 127.0.0.1:46920
I20250905 07:50:31.669857 3184 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "default.Table_c6a86ab58c8f4eddbf3e0c4edf058247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:31.668823 3186 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "default.Table_4907c91834874a1a88a3312b6a0be96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:31.670822 3183 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "default.Table_cd8e6f84c6614cd58e55c59c90383205"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:31.671412 3182 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "default.Table_b3cf8c80578f4b4f84327aedafee6a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:31.672295 3181 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
I20250905 07:50:31.672534 3181 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
I20250905 07:50:31.673586 3180 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:31.673831 3180 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:31.675024 3179 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "default.Table_e3daa55fc9e5447f82536064de82c58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250905 07:50:31.677714 3241 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250905 07:50:31.680599 3188 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Sending DeleteTablet for 3 replicas of tablet 9e2858d296ec4022a337e6f3d6fa629d
@ 0x7fd51ef56dcd google::LogMessage::Fail() at ??:0
I20250905 07:50:31.682602 3187 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Sending DeleteTablet for 3 replicas of tablet 8b52a829672943da8121d121052697ab
I20250905 07:50:31.683455 3187 catalog_manager.cc:6652] ReplaceTablet: tablet 8b52a829672943da8121d121052697ab deleted and replaced by tablet 56460f95360a413b8b3cb31195d4f102
@ 0x7fd51ef5ab93 google::LogMessage::SendToLog() at ??:0
I20250905 07:50:31.687137 3188 catalog_manager.cc:6652] ReplaceTablet: tablet 9e2858d296ec4022a337e6f3d6fa629d deleted and replaced by tablet f08841e4d54446699cd387180cdd871a
I20250905 07:50:31.687541 3188 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:31.687836 3188 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46920:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:31.688036 3187 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "default.Table_7c0359e055804d0b9a84bd4dd5e60634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fd51ef567cc google::LogMessage::Flush() at ??:0
@ 0x7fd51ef56a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd5353d27b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250905 07:50:31.712617 2320 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b52a829672943da8121d121052697ab with delete_type TABLET_DATA_DELETED (replaced by tablet 56460f95360a413b8b3cb31195d4f102) from {username='slave'} at 127.0.0.1:35534
I20250905 07:50:31.713104 2318 tablet_service.cc:1468] Processing CreateTablet for tablet 56460f95360a413b8b3cb31195d4f102 (DEFAULT_TABLE table=default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:31.713582 2318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56460f95360a413b8b3cb31195d4f102. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:31.713814 2319 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e2858d296ec4022a337e6f3d6fa629d with delete_type TABLET_DATA_DELETED (replaced by tablet f08841e4d54446699cd387180cdd871a) from {username='slave'} at 127.0.0.1:35534
I20250905 07:50:31.714393 2317 tablet_service.cc:1468] Processing CreateTablet for tablet f08841e4d54446699cd387180cdd871a (DEFAULT_TABLE table=default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:31.714841 2317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08841e4d54446699cd387180cdd871a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd5353d1061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250905 07:50:31.721459 2599 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b52a829672943da8121d121052697ab with delete_type TABLET_DATA_DELETED (replaced by tablet 56460f95360a413b8b3cb31195d4f102) from {username='slave'} at 127.0.0.1:33330
I20250905 07:50:31.721765 2463 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e2858d296ec4022a337e6f3d6fa629d with delete_type TABLET_DATA_DELETED (replaced by tablet f08841e4d54446699cd387180cdd871a) from {username='slave'} at 127.0.0.1:43704
I20250905 07:50:31.721837 2600 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e2858d296ec4022a337e6f3d6fa629d with delete_type TABLET_DATA_DELETED (replaced by tablet f08841e4d54446699cd387180cdd871a) from {username='slave'} at 127.0.0.1:33330
I20250905 07:50:31.722376 2601 tablet_service.cc:1468] Processing CreateTablet for tablet 56460f95360a413b8b3cb31195d4f102 (DEFAULT_TABLE table=default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:31.722514 2460 tablet_service.cc:1468] Processing CreateTablet for tablet f08841e4d54446699cd387180cdd871a (DEFAULT_TABLE table=default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:31.722887 2601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56460f95360a413b8b3cb31195d4f102. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:31.722999 2460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08841e4d54446699cd387180cdd871a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd5353d8cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fd5353d8b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250905 07:50:31.728654 2462 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b52a829672943da8121d121052697ab with delete_type TABLET_DATA_DELETED (replaced by tablet 56460f95360a413b8b3cb31195d4f102) from {username='slave'} at 127.0.0.1:43704
I20250905 07:50:31.729394 2598 tablet_service.cc:1468] Processing CreateTablet for tablet f08841e4d54446699cd387180cdd871a (DEFAULT_TABLE table=default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0xa774dd std::function<>::operator()() at ??:?
I20250905 07:50:31.729934 2598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08841e4d54446699cd387180cdd871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:31.732421 2461 tablet_service.cc:1468] Processing CreateTablet for tablet 56460f95360a413b8b3cb31195d4f102 (DEFAULT_TABLE table=default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:31.732956 2461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56460f95360a413b8b3cb31195d4f102. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd521cbf2b9 kudu::Thread::SuperviseThread() at ??:0
I20250905 07:50:31.742794 3412 tablet_replica.cc:331] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:31.743077 3412 raft_consensus.cc:2241] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:31.743381 3412 raft_consensus.cc:2270] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:31.743729 3413 tablet_replica.cc:331] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:31.743955 3413 raft_consensus.cc:2241] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:50:31.744088 3412 ts_tablet_manager.cc:1905] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:31.744681 3413 raft_consensus.cc:2270] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:31.745290 3413 ts_tablet_manager.cc:1905] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fd5251166db start_thread at ??:0
@ 0x7fd51de0771f clone at ??:0
I20250905 07:50:31.751008 3417 tablet_bootstrap.cc:492] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:31.755553 3412 ts_tablet_manager.cc:1918] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:31.755736 3412 log.cc:1199] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/8b52a829672943da8121d121052697ab
I20250905 07:50:31.756281 3412 ts_tablet_manager.cc:1939] T 8b52a829672943da8121d121052697ab P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:31.756664 3416 tablet_bootstrap.cc:492] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:31.757615 3412 tablet_replica.cc:331] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:31.757879 3412 raft_consensus.cc:2241] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:31.758070 3412 raft_consensus.cc:2270] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:31.758657 3412 ts_tablet_manager.cc:1905] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:31.759303 3415 tablet_replica.cc:331] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:31.759570 3415 raft_consensus.cc:2241] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:50:31.760120 3417 tablet_bootstrap.cc:654] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:31.761071 3416 tablet_bootstrap.cc:654] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:31.767146 3412 ts_tablet_manager.cc:1918] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:31.767336 3412 log.cc:1199] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/9e2858d296ec4022a337e6f3d6fa629d
I20250905 07:50:31.767810 3412 ts_tablet_manager.cc:1939] T 9e2858d296ec4022a337e6f3d6fa629d P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:31.772430 3413 ts_tablet_manager.cc:1918] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:31.772603 3413 log.cc:1199] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/8b52a829672943da8121d121052697ab
I20250905 07:50:31.773124 3413 ts_tablet_manager.cc:1939] T 8b52a829672943da8121d121052697ab P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:31.774438 3413 tablet_replica.cc:331] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:31.774670 3413 raft_consensus.cc:2241] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:31.774850 3413 raft_consensus.cc:2270] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:31.775434 3413 ts_tablet_manager.cc:1905] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:31.776734 3414 tablet_bootstrap.cc:492] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:31.777279 3417 tablet_bootstrap.cc:492] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:31.777420 3417 ts_tablet_manager.cc:1397] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.006s
I20250905 07:50:31.778285 3417 raft_consensus.cc:357] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.778489 3417 raft_consensus.cc:383] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:31.778584 3417 raft_consensus.cc:738] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:31.778836 3414 tablet_bootstrap.cc:654] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:31.778867 3417 consensus_queue.cc:260] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.782778 3417 ts_tablet_manager.cc:1428] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250905 07:50:31.783851 3416 tablet_bootstrap.cc:492] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:31.784073 3416 ts_tablet_manager.cc:1397] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20250905 07:50:31.785147 3416 raft_consensus.cc:357] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.785408 3416 raft_consensus.cc:383] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:31.785511 3416 raft_consensus.cc:738] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:31.785805 3416 consensus_queue.cc:260] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.788404 3415 raft_consensus.cc:2270] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:31.788770 3413 ts_tablet_manager.cc:1918] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:31.788933 3413 log.cc:1199] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/9e2858d296ec4022a337e6f3d6fa629d
I20250905 07:50:31.789165 3415 ts_tablet_manager.cc:1905] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:31.789376 3413 ts_tablet_manager.cc:1939] T 9e2858d296ec4022a337e6f3d6fa629d P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:31.789885 3416 ts_tablet_manager.cc:1428] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 07:50:31.797384 3417 tablet_bootstrap.cc:492] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:31.799749 3416 tablet_bootstrap.cc:492] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:31.802129 3416 tablet_bootstrap.cc:654] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:31.804937 3417 tablet_bootstrap.cc:654] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:31.806825 3414 tablet_bootstrap.cc:492] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:31.806905 3416 tablet_bootstrap.cc:492] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:31.806984 3414 ts_tablet_manager.cc:1397] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.013s
I20250905 07:50:31.807056 3416 ts_tablet_manager.cc:1397] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250905 07:50:31.807163 3415 ts_tablet_manager.cc:1918] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:31.807318 3415 log.cc:1199] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/9e2858d296ec4022a337e6f3d6fa629d
I20250905 07:50:31.807888 3415 ts_tablet_manager.cc:1939] T 9e2858d296ec4022a337e6f3d6fa629d P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:31.807833 3416 raft_consensus.cc:357] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.808071 3416 raft_consensus.cc:383] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:31.808175 3416 raft_consensus.cc:738] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:31.807845 3414 raft_consensus.cc:357] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.808568 3414 raft_consensus.cc:383] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:31.808486 3416 consensus_queue.cc:260] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.809001 3416 ts_tablet_manager.cc:1428] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:31.809621 3415 tablet_replica.cc:331] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:31.809877 3415 raft_consensus.cc:2241] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:31.810050 3415 raft_consensus.cc:2270] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:31.810683 3415 ts_tablet_manager.cc:1905] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:31.810954 3417 tablet_bootstrap.cc:492] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:31.811110 3417 ts_tablet_manager.cc:1397] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250905 07:50:31.811856 3417 raft_consensus.cc:357] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.812112 3417 raft_consensus.cc:383] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:31.812191 3417 raft_consensus.cc:738] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:31.812448 3417 consensus_queue.cc:260] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.812988 3417 ts_tablet_manager.cc:1428] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 07:50:31.808663 3414 raft_consensus.cc:738] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:31.813427 3414 consensus_queue.cc:260] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.814045 3414 ts_tablet_manager.cc:1428] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 07:50:31.817148 3414 tablet_bootstrap.cc:492] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:31.820628 3414 tablet_bootstrap.cc:654] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:31.822520 3414 tablet_bootstrap.cc:492] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:31.822685 3414 ts_tablet_manager.cc:1397] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250905 07:50:31.823454 3415 ts_tablet_manager.cc:1918] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:31.823438 3414 raft_consensus.cc:357] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.823599 3415 log.cc:1199] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/8b52a829672943da8121d121052697ab
I20250905 07:50:31.823678 3414 raft_consensus.cc:383] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:31.823781 3414 raft_consensus.cc:738] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:31.824075 3414 consensus_queue.cc:260] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:31.824576 3414 ts_tablet_manager.cc:1428] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 07:50:31.826880 3415 ts_tablet_manager.cc:1939] T 8b52a829672943da8121d121052697ab P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
W20250905 07:50:31.968838 2552 connection.cc:537] server connection from 127.0.0.1:33330 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:31.968895 2419 connection.cc:537] server connection from 127.0.0.1:43704 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:31.968881 2276 connection.cc:537] server connection from 127.0.0.1:35534 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:31.969180 2552 connection.cc:537] client connection to 127.1.178.189:43391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:31.969602 2273 connection.cc:537] client connection to 127.1.178.189:43391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:31.969712 2673 connection.cc:537] client connection to 127.1.178.189:43391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:31.970357 2673 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.189:43391)
W20250905 07:50:31.971014 2673 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.189:43391)
W20250905 07:50:31.971213 2673 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.189:43391)
W20250905 07:50:31.971905 2673 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.189:43391)
I20250905 07:50:32.118541 3389 raft_consensus.cc:491] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:32.118742 3389 raft_consensus.cc:513] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:32.119450 3389 leader_election.cc:290] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:32.120064 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56460f95360a413b8b3cb31195d4f102" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:32.120076 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56460f95360a413b8b3cb31195d4f102" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:32.120265 2619 raft_consensus.cc:2466] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:32.120289 2483 raft_consensus.cc:2466] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 0.
I20250905 07:50:32.120748 2273 leader_election.cc:304] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:32.121016 3389 raft_consensus.cc:2802] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:32.121140 3389 raft_consensus.cc:491] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:32.121235 3389 raft_consensus.cc:3058] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:32.123104 3389 raft_consensus.cc:513] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:32.123600 3389 leader_election.cc:290] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Requested vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:32.124122 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56460f95360a413b8b3cb31195d4f102" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:32.124187 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56460f95360a413b8b3cb31195d4f102" candidate_uuid: "b902ef48db944a02813b8543b2f91a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:32.124356 2619 raft_consensus.cc:3058] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:32.124375 2483 raft_consensus.cc:3058] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:32.127192 2619 raft_consensus.cc:2466] T 56460f95360a413b8b3cb31195d4f102 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:32.127180 2483 raft_consensus.cc:2466] T 56460f95360a413b8b3cb31195d4f102 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902ef48db944a02813b8543b2f91a7e in term 1.
I20250905 07:50:32.127607 2273 leader_election.cc:304] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:32.127974 3389 raft_consensus.cc:2802] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:32.128923 3389 raft_consensus.cc:695] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: Becoming Leader. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Running, Role: LEADER
I20250905 07:50:32.129489 3389 consensus_queue.cc:237] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:32.196410 3437 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:32.196689 3437 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:32.198467 3437 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:32.199420 3311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:32.200860 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:32.201078 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:32.201185 3311 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:32.201594 3437 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 07:50:32.301291 3010 raft_consensus.cc:491] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:32.301503 3010 raft_consensus.cc:513] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:32.302172 3010 leader_election.cc:290] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:32.302814 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08841e4d54446699cd387180cdd871a" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:32.302824 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08841e4d54446699cd387180cdd871a" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:32.303037 2619 raft_consensus.cc:2466] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:32.303050 2340 raft_consensus.cc:2466] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:32.303449 2416 leader_election.cc:304] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:32.303716 3010 raft_consensus.cc:2802] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:32.303843 3010 raft_consensus.cc:491] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:32.303958 3010 raft_consensus.cc:3058] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:32.306078 3010 raft_consensus.cc:513] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:32.306772 3010 leader_election.cc:290] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Requested vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:32.307191 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08841e4d54446699cd387180cdd871a" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:32.307265 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08841e4d54446699cd387180cdd871a" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:32.307379 2619 raft_consensus.cc:3058] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:32.307482 2340 raft_consensus.cc:3058] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:32.309253 2340 raft_consensus.cc:2466] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:32.309211 2619 raft_consensus.cc:2466] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:32.309672 2416 leader_election.cc:304] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:32.309957 3010 raft_consensus.cc:2802] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:32.310338 3010 raft_consensus.cc:695] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Running, Role: LEADER
I20250905 07:50:32.310642 3010 consensus_queue.cc:237] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:32.866321 3446 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:32.866554 3446 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:32.867304 3446 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:32.869242 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:32.869400 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:32.869488 3311 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:32.869745 3446 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 07:50:33.052558 3389 consensus_queue.cc:1035] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:33.058109 3451 consensus_queue.cc:1035] T 56460f95360a413b8b3cb31195d4f102 P b902ef48db944a02813b8543b2f91a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:33.216972 3010 consensus_queue.cc:1035] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:33.223009 3459 consensus_queue.cc:1035] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:33.608067 3461 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:33.608276 3461 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:33.608937 3461 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
W20250905 07:50:33.611435 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:33.611696 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:33.611797 3311 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81e2a8aedb144e61891de4dcd390ff35; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:33.612138 3461 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 07:50:33.951797 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--webserver_interface=127.1.178.189
--webserver_port=37293
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:34.291493 3468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:34.291939 3468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:34.292093 3468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:34.300803 3468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:34.300897 3468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:34.300953 3468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:34.301003 3468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:34.312150 3468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.178.189
--webserver_port=37293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:34.313745 3468 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:34.315478 3468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:34.323212 3481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:34.323892 3482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:34.324515 3484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:34.324982 3468 server_base.cc:1047] running on GCE node
I20250905 07:50:34.325683 3468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:34.327006 3468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:34.328207 3468 hybrid_clock.cc:648] HybridClock initialized: now 1757058634328155 us; error 43 us; skew 500 ppm
I20250905 07:50:34.328557 3468 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:34.330791 3468 webserver.cc:480] Webserver started at http://127.1.178.189:37293/ using document root <none> and password file <none>
I20250905 07:50:34.331543 3468 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:34.331655 3468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:34.337270 3468 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20250905 07:50:34.340153 3490 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:34.341491 3468 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 07:50:34.341650 3468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
uuid: "65033f3e4b44455389e29a5e1c9f82cd"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:34.342320 3468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:34.359077 3468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:34.359871 3468 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:34.360162 3468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:34.381561 3468 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.189:43391
I20250905 07:50:34.381582 3541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.189:43391 every 8 connection(s)
I20250905 07:50:34.383176 3468 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
I20250905 07:50:34.383538 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 3468
I20250905 07:50:34.390339 3542 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:34.393869 3542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap starting.
I20250905 07:50:34.406174 3542 log.cc:826] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:34.442618 3542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=30 ignored=0} mutations{seen=32 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:50:34.443188 3542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap complete.
I20250905 07:50:34.448840 3542 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:34.449697 3542 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Initialized, Role: FOLLOWER
I20250905 07:50:34.450388 3542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 8.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:34.451144 3546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:34.451391 3546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:34.452252 3542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:34.480266 3560 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:34.480494 3560 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: loading cluster ID for follower catalog manager: success
I20250905 07:50:34.485041 3560 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: acquiring CA information for follower catalog manager: success
I20250905 07:50:34.488220 3560 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:34.487897 3468 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:50:34.492446 3468 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:34.680112 2387 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:34.684243 3507 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:48221; Asking this server to re-register.
I20250905 07:50:34.685106 2387 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:34.687881 3507 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:34.725462 3566 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:34.725651 3566 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:34.726418 3566 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
I20250905 07:50:34.737762 2668 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:34.739733 3507 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:57665; Asking this server to re-register.
I20250905 07:50:34.740559 2668 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:34.742259 3507 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
W20250905 07:50:34.745424 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:34.745564 3517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:34.746093 3517 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 8.
I20250905 07:50:34.746814 3311 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:34.747139 3566 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 07:50:34.747252 3566 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:34.747356 3566 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 8 FOLLOWER]: Advancing to term 9
I20250905 07:50:34.749594 3566 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:34.750146 3566 leader_election.cc:290] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 election: Requested vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391)
I20250905 07:50:34.751049 3517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81e2a8aedb144e61891de4dcd390ff35" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:34.751266 3517 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 8 FOLLOWER]: Advancing to term 9
W20250905 07:50:34.752434 3311 leader_election.cc:336] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:34.757380 3517 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e2a8aedb144e61891de4dcd390ff35 in term 9.
I20250905 07:50:34.757854 3311 leader_election.cc:304] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35; no voters: 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:34.758159 3566 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 9 FOLLOWER]: Leader election won for term 9
I20250905 07:50:34.758639 3566 raft_consensus.cc:695] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [term 9 LEADER]: Becoming Leader. State: Replica: 81e2a8aedb144e61891de4dcd390ff35, State: Running, Role: LEADER
I20250905 07:50:34.759079 3566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 8.45, Last appended by leader: 45, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:34.760946 3571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 9 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:34.761258 3571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:34.761895 3573 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:34.764535 3573 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:34.765108 3573 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:34.765484 3573 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:34.765877 3573 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:34.766216 3573 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:34.766561 3573 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:34.766974 3573 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:34.767464 3573 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:34.767860 3573 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:34.771106 3573 tablet_loader.cc:96] loaded metadata for tablet 0c28568a8557410c9104026b8f1cf3f1 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:34.772032 3573 tablet_loader.cc:96] loaded metadata for tablet 56460f95360a413b8b3cb31195d4f102 (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:34.772430 3573 tablet_loader.cc:96] loaded metadata for tablet 7a4ef48a5e67406297cf21413236f563 (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:34.772832 3573 tablet_loader.cc:96] loaded metadata for tablet 84726ef69a6742a9b046a8833e406590 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:34.773478 3573 tablet_loader.cc:96] loaded metadata for tablet 95be079f58414b529a5f2009f56bb009 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:34.774267 3573 tablet_loader.cc:96] loaded metadata for tablet c022e7ee959d4d19958437a1ed855bbd (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:34.774744 3573 tablet_loader.cc:96] loaded metadata for tablet f08841e4d54446699cd387180cdd871a (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:34.775297 3573 tablet_loader.cc:96] loaded metadata for tablet f7fceab9655d4d9ea0e142f0e3a4d06e (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:34.775663 3573 tablet_loader.cc:96] loaded metadata for tablet ff5338af10954f3f9ee797c37a1ca9cc (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:34.775957 3573 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:34.777350 3573 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:34.777439 3573 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:34.779371 3573 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:34.780766 3573 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Loaded TSK: 0
I20250905 07:50:34.781864 3573 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:34.782773 3573 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:34.783648 3573 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:34.800909 3327 master_service.cc:948] ReplaceTablet: received request to replace tablet 3743b39493ed452e89b1e3223bb1ba88 from {username='slave'} at 127.0.0.1:44014
I20250905 07:50:34.801522 3326 master_service.cc:948] ReplaceTablet: received request to replace tablet 84726ef69a6742a9b046a8833e406590 from {username='slave'} at 127.0.0.1:44014
I20250905 07:50:34.802572 3318 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44014:
table { table_name: "default.Table_a908ec5ae1aa46e8b872ff681d610476" } new_table_name: "default.Table_23be441c886b4e17929155f056fdb23b" modify_external_catalogs: true
I20250905 07:50:34.803581 3321 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44014:
name: "default.Table_8f62349a373e45b8ae063dafbfc3cdcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:34.804193 3320 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44014:
name: "default.Table_06530af7099e45c7b150408f8c1a3436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:34.803642 3319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44014:
name: "default.Table_6abbacd07c2f4ab4b74761cd100dcf58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:34.804610 3322 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44014:
table { table_name: "default.Table_fbd6c4d89fd54fdd9765576cc3cf5cbc" } new_table_name: "default.Table_68b9ab7fcb3245ad84e0fe0e5c05d746" modify_external_catalogs: true
I20250905 07:50:34.805184 3323 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c28568a8557410c9104026b8f1cf3f1 from {username='slave'} at 127.0.0.1:44014
I20250905 07:50:34.806193 3325 master_service.cc:948] ReplaceTablet: received request to replace tablet 083c74057a6945288b259678e0ff2223 from {username='slave'} at 127.0.0.1:44014
I20250905 07:50:34.806972 3325 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44014:
name: "default.Table_c6a86ab58c8f4eddbf3e0c4edf058247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:34.807441 3324 master_service.cc:948] ReplaceTablet: received request to replace tablet 84726ef69a6742a9b046a8833e406590 from {username='slave'} at 127.0.0.1:44014
I20250905 07:50:34.808223 3327 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44014:
table { table_name: "default.Table_146568e744294fdd8f082cdbd8451039" } new_table_name: "default.Table_bcc153425f724fa0aee1ba610df560ce" modify_external_catalogs: true
I20250905 07:50:34.813530 3517 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Refusing update from remote peer 81e2a8aedb144e61891de4dcd390ff35: Log matching property violated. Preceding OpId in replica: term: 8 index: 45. Preceding OpId from leader: term: 9 index: 47. (index mismatch)
I20250905 07:50:34.815398 3571 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
W20250905 07:50:34.817083 3311 consensus_peers.cc:489] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 -> Peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Couldn't send request to peer 9dbf15725a7342da830f57c0535eb72d. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250905 07:50:34.820940 3380 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250905 07:50:34.821821 2530 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
@ 0x7f800d513dcd google::LogMessage::Fail() at ??:0
@ 0x7f800d517b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f800d5137cc google::LogMessage::Flush() at ??:0
@ 0x7f800d513a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f802398f7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250905 07:50:34.846005 3507 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:45123; Asking this server to re-register.
I20250905 07:50:34.846040 3546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e2a8aedb144e61891de4dcd390ff35. Latest consensus state: current_term: 9 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:34.846334 3546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:34.846906 2530 heartbeater.cc:461] Registering TS with master...
@ 0x7f802398e061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f8023995cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250905 07:50:34.864120 3323 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Sending DeleteTablet for 3 replicas of tablet 0c28568a8557410c9104026b8f1cf3f1
I20250905 07:50:34.866308 3323 catalog_manager.cc:6652] ReplaceTablet: tablet 0c28568a8557410c9104026b8f1cf3f1 deleted and replaced by tablet 2804dc5bb8f940cca6515831b9b30c72
I20250905 07:50:34.867869 3326 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35: Sending DeleteTablet for 3 replicas of tablet 84726ef69a6742a9b046a8833e406590
I20250905 07:50:34.868432 3326 catalog_manager.cc:6652] ReplaceTablet: tablet 84726ef69a6742a9b046a8833e406590 deleted and replaced by tablet a6d924df2b9244feb4813bafa9a218b3
I20250905 07:50:34.883772 3546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:34.884047 3546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:34.884112 3507 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:34.884166 3323 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44014:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:34.884430 3323 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44014:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
@ 0x7f8023995b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250905 07:50:34.889515 3324 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44014:
name: "default.Table_cd8e6f84c6614cd58e55c59c90383205"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:34.890393 3326 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44014:
name: "default.Table_4907c91834874a1a88a3312b6a0be96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0xa774dd std::function<>::operator()() at ??:?
I20250905 07:50:34.892238 3571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "81e2a8aedb144e61891de4dcd390ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:34.892510 3571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e2a8aedb144e61891de4dcd390ff35 [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:34.897630 2460 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28568a8557410c9104026b8f1cf3f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2804dc5bb8f940cca6515831b9b30c72) from {username='slave'} at 127.0.0.1:42750
I20250905 07:50:34.897967 2318 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28568a8557410c9104026b8f1cf3f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2804dc5bb8f940cca6515831b9b30c72) from {username='slave'} at 127.0.0.1:34244
I20250905 07:50:34.898113 2317 tablet_service.cc:1515] Processing DeleteTablet for tablet 84726ef69a6742a9b046a8833e406590 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d924df2b9244feb4813bafa9a218b3) from {username='slave'} at 127.0.0.1:34244
I20250905 07:50:34.898530 2463 tablet_service.cc:1515] Processing DeleteTablet for tablet 84726ef69a6742a9b046a8833e406590 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d924df2b9244feb4813bafa9a218b3) from {username='slave'} at 127.0.0.1:42750
I20250905 07:50:34.900049 2601 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28568a8557410c9104026b8f1cf3f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2804dc5bb8f940cca6515831b9b30c72) from {username='slave'} at 127.0.0.1:48580
I20250905 07:50:34.900477 2598 tablet_service.cc:1515] Processing DeleteTablet for tablet 84726ef69a6742a9b046a8833e406590 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d924df2b9244feb4813bafa9a218b3) from {username='slave'} at 127.0.0.1:48580
@ 0x7f801027c2b9 kudu::Thread::SuperviseThread() at ??:0
I20250905 07:50:34.903118 2598 tablet_service.cc:1468] Processing CreateTablet for tablet 2804dc5bb8f940cca6515831b9b30c72 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:34.903596 2598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2804dc5bb8f940cca6515831b9b30c72. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f80136d36db start_thread at ??:0
@ 0x7f800c3c471f clone at ??:0
I20250905 07:50:34.907389 2317 tablet_service.cc:1468] Processing CreateTablet for tablet 2804dc5bb8f940cca6515831b9b30c72 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:34.907523 2463 tablet_service.cc:1468] Processing CreateTablet for tablet 2804dc5bb8f940cca6515831b9b30c72 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:34.907778 2317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2804dc5bb8f940cca6515831b9b30c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:34.908077 2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2804dc5bb8f940cca6515831b9b30c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:34.914781 3598 tablet_replica.cc:331] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:34.915022 3598 raft_consensus.cc:2241] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:34.915309 3598 raft_consensus.cc:2270] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:34.916085 3598 ts_tablet_manager.cc:1905] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:34.916545 3596 tablet_replica.cc:331] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:34.916734 3596 raft_consensus.cc:2241] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:34.916944 3596 raft_consensus.cc:2270] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:34.917501 3596 ts_tablet_manager.cc:1905] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:34.920997 3599 tablet_bootstrap.cc:492] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:34.921367 3597 tablet_replica.cc:331] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:34.921552 3597 raft_consensus.cc:2241] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:50:34.922082 3597 raft_consensus.cc:2270] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:34.922688 3597 ts_tablet_manager.cc:1905] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:34.923005 3599 tablet_bootstrap.cc:654] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:34.923401 3601 tablet_bootstrap.cc:492] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:34.925467 3601 tablet_bootstrap.cc:654] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:34.928716 3600 tablet_bootstrap.cc:492] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:34.930979 3600 tablet_bootstrap.cc:654] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:34.931214 3599 tablet_bootstrap.cc:492] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:34.931381 3599 ts_tablet_manager.cc:1397] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250905 07:50:34.931735 3596 ts_tablet_manager.cc:1918] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:34.931886 3596 log.cc:1199] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/0c28568a8557410c9104026b8f1cf3f1
I20250905 07:50:34.932463 3596 ts_tablet_manager.cc:1939] T 0c28568a8557410c9104026b8f1cf3f1 P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:34.932776 3599 raft_consensus.cc:357] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:34.932964 3599 raft_consensus.cc:383] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:34.933043 3599 raft_consensus.cc:738] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:34.933252 3599 consensus_queue.cc:260] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:34.933897 3596 tablet_replica.cc:331] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:34.934175 3596 raft_consensus.cc:2241] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:34.934440 3596 raft_consensus.cc:2270] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:34.935151 3596 ts_tablet_manager.cc:1905] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:34.938108 3599 ts_tablet_manager.cc:1428] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 07:50:34.941679 3596 ts_tablet_manager.cc:1918] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:34.941839 3596 log.cc:1199] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/84726ef69a6742a9b046a8833e406590
I20250905 07:50:34.942363 3596 ts_tablet_manager.cc:1939] T 84726ef69a6742a9b046a8833e406590 P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:34.944128 3597 ts_tablet_manager.cc:1918] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:34.944252 3597 log.cc:1199] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/0c28568a8557410c9104026b8f1cf3f1
I20250905 07:50:34.944737 3597 ts_tablet_manager.cc:1939] T 0c28568a8557410c9104026b8f1cf3f1 P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:34.945502 3597 tablet_replica.cc:331] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:34.945667 3597 raft_consensus.cc:2241] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:50:34.945892 3597 raft_consensus.cc:2270] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:34.946364 3597 ts_tablet_manager.cc:1905] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:34.948025 3598 ts_tablet_manager.cc:1918] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:34.948179 3598 log.cc:1199] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/0c28568a8557410c9104026b8f1cf3f1
I20250905 07:50:34.948845 3598 ts_tablet_manager.cc:1939] T 0c28568a8557410c9104026b8f1cf3f1 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:34.949047 3601 tablet_bootstrap.cc:492] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:34.949178 3601 ts_tablet_manager.cc:1397] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.006s
I20250905 07:50:34.950027 3601 raft_consensus.cc:357] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:34.950212 3601 raft_consensus.cc:383] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:34.950311 3601 raft_consensus.cc:738] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:34.950714 3598 tablet_replica.cc:331] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:34.950641 3601 consensus_queue.cc:260] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:34.950917 3598 raft_consensus.cc:2241] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:34.951079 3598 raft_consensus.cc:2270] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:34.951104 2528 heartbeater.cc:499] Master 127.1.178.188:39743 was elected leader, sending a full tablet report...
I20250905 07:50:34.951628 3598 ts_tablet_manager.cc:1905] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:34.953795 3601 ts_tablet_manager.cc:1428] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 07:50:34.966984 3600 tablet_bootstrap.cc:492] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:34.967374 3600 ts_tablet_manager.cc:1397] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.000s
I20250905 07:50:34.967523 3597 ts_tablet_manager.cc:1918] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:34.967705 3597 log.cc:1199] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/84726ef69a6742a9b046a8833e406590
I20250905 07:50:34.968163 3597 ts_tablet_manager.cc:1939] T 84726ef69a6742a9b046a8833e406590 P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:34.968425 3600 raft_consensus.cc:357] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:34.968658 3600 raft_consensus.cc:383] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:34.968770 3600 raft_consensus.cc:738] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:34.969077 3600 consensus_queue.cc:260] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:34.972381 3600 ts_tablet_manager.cc:1428] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 07:50:34.974898 3598 ts_tablet_manager.cc:1918] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:34.975035 3598 log.cc:1199] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/84726ef69a6742a9b046a8833e406590
I20250905 07:50:34.975457 3598 ts_tablet_manager.cc:1939] T 84726ef69a6742a9b046a8833e406590 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
W20250905 07:50:35.113082 2273 connection.cc:537] server connection from 127.0.0.1:34244 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:35.113118 2554 connection.cc:537] server connection from 127.0.0.1:48580 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:35.113358 2554 connection.cc:537] client connection to 127.1.178.188:39743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:35.113615 2275 connection.cc:537] client connection to 127.1.178.188:39743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:35.113664 2675 connection.cc:537] client connection to 127.1.178.188:39743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:35.114812 2675 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.188:39743)
W20250905 07:50:35.114933 2418 connection.cc:537] server connection from 127.0.0.1:42750 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:35.115154 2418 connection.cc:537] client connection to 127.1.178.188:39743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:35.115419 2675 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.188:39743)
W20250905 07:50:35.115761 2675 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.188:39743)
W20250905 07:50:35.115971 2675 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.188:39743)
I20250905 07:50:35.161199 2957 raft_consensus.cc:491] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:35.161441 2957 raft_consensus.cc:513] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:35.162303 2957 leader_election.cc:290] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:35.162760 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2804dc5bb8f940cca6515831b9b30c72" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:35.162822 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2804dc5bb8f940cca6515831b9b30c72" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:35.162981 2483 raft_consensus.cc:2466] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:35.163043 2340 raft_consensus.cc:2466] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:35.163452 2552 leader_election.cc:304] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:35.163713 2957 raft_consensus.cc:2802] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:35.163811 2957 raft_consensus.cc:491] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:35.163892 2957 raft_consensus.cc:3058] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:35.165781 2957 raft_consensus.cc:513] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:35.166280 2957 leader_election.cc:290] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Requested vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:35.166750 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2804dc5bb8f940cca6515831b9b30c72" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:35.166911 2483 raft_consensus.cc:3058] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:35.166859 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2804dc5bb8f940cca6515831b9b30c72" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:35.167049 2340 raft_consensus.cc:3058] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:35.168877 2483 raft_consensus.cc:2466] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:35.168962 2340 raft_consensus.cc:2466] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:35.169318 2552 leader_election.cc:304] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4; no voters:
I20250905 07:50:35.169612 2957 raft_consensus.cc:2802] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:35.170003 2957 raft_consensus.cc:695] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Becoming Leader. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Running, Role: LEADER
I20250905 07:50:35.170331 2957 consensus_queue.cc:237] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } }
I20250905 07:50:35.399596 3611 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:35.399850 3611 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:35.402168 3611 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:35.404763 3491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:35.406195 3491 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:35.406808 3493 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:35.406965 3493 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:35.407296 3611 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250905 07:50:36.064679 3619 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:36.064885 3619 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:36.065708 3619 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:36.067677 3491 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:36.067939 3493 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:36.068065 3493 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:36.068300 3619 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250905 07:50:36.148608 2957 consensus_queue.cc:1035] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:36.160169 2957 consensus_queue.cc:1035] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:36.826638 3633 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:36.826854 3633 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:36.827508 3633 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:36.829690 3491 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:36.829890 3493 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:36.830006 3493 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:36.830252 3633 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250905 07:50:37.710457 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--webserver_interface=127.1.178.190
--webserver_port=34789
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:38.063082 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:38.063465 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:38.063547 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:38.072496 3646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:38.072613 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:38.072671 3646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:38.072722 3646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:38.083159 3646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.178.190
--webserver_port=34789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:38.085026 3646 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:38.087339 3646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:38.096426 3654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:38.096347 3656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:38.096388 3653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:38.097494 3646 server_base.cc:1047] running on GCE node
I20250905 07:50:38.098403 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:38.100085 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:38.101308 3646 hybrid_clock.cc:648] HybridClock initialized: now 1757058638101251 us; error 53 us; skew 500 ppm
I20250905 07:50:38.101642 3646 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:38.104074 3646 webserver.cc:480] Webserver started at http://127.1.178.190:34789/ using document root <none> and password file <none>
I20250905 07:50:38.104831 3646 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:38.104931 3646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:38.110791 3646 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 07:50:38.114012 3662 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:38.115523 3646 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 07:50:38.115733 3646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
uuid: "9dbf15725a7342da830f57c0535eb72d"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:38.116694 3646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:38.132253 3646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:38.133124 3646 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:38.133419 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:38.153018 3646 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.190:35357
I20250905 07:50:38.153043 3713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.190:35357 every 8 connection(s)
I20250905 07:50:38.154754 3646 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
I20250905 07:50:38.160830 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 3646
I20250905 07:50:38.161880 3714 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:38.166991 3714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap starting.
I20250905 07:50:38.186586 3714 log.cc:826] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:38.198869 2529 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:38.199174 2386 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:38.223007 3714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap replayed 1/1 log segments. Stats: ops{read=41 overwritten=0 applied=40 ignored=0} inserts{seen=28 ignored=0} mutations{seen=24 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 07:50:38.223629 3714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap complete.
I20250905 07:50:38.229393 3714 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:38.232065 3714 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Initialized, Role: FOLLOWER
I20250905 07:50:38.232771 3714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 7.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:38.233500 3723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:38.233742 3723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:38.234635 3714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:38.247887 3737 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:38.248173 3737 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: loading cluster ID for follower catalog manager: success
I20250905 07:50:38.255481 3737 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: acquiring CA information for follower catalog manager: success
I20250905 07:50:38.259091 3737 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:38.273003 2667 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:38.276152 3679 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:46357; Asking this server to re-register.
I20250905 07:50:38.276978 2667 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:38.279086 3679 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:38.280113 3646 master_runner.cc:186] Error getting master registration for 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111),
I20250905 07:50:38.285775 3646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:38.290066 3745 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:38.290285 3745 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:38.291183 3745 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:38.293326 3493 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:38.299180 3689 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d" is_pre_election: true
I20250905 07:50:38.299813 3689 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65033f3e4b44455389e29a5e1c9f82cd in term 7.
I20250905 07:50:38.300773 3491 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:38.301280 3745 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250905 07:50:38.301483 3745 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Starting leader election (detected failure of leader 81e2a8aedb144e61891de4dcd390ff35)
I20250905 07:50:38.301599 3745 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 9 FOLLOWER]: Advancing to term 10
I20250905 07:50:38.304787 3745 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:38.305660 3745 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 election: Requested vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:38.306047 3689 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65033f3e4b44455389e29a5e1c9f82cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "9dbf15725a7342da830f57c0535eb72d"
I20250905 07:50:38.306278 3689 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 7 FOLLOWER]: Advancing to term 10
W20250905 07:50:38.307350 3493 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:38.310250 3689 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65033f3e4b44455389e29a5e1c9f82cd in term 10.
I20250905 07:50:38.310724 3491 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:38.311024 3745 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 10 FOLLOWER]: Leader election won for term 10
I20250905 07:50:38.311465 3745 raft_consensus.cc:695] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 10 LEADER]: Becoming Leader. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Running, Role: LEADER
I20250905 07:50:38.311908 3745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 9.49, Last appended by leader: 49, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:38.313632 3749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65033f3e4b44455389e29a5e1c9f82cd. Latest consensus state: current_term: 10 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:38.313951 3749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:38.314657 3751 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:38.317293 3751 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:38.317724 3751 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:38.318089 3751 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:38.318442 3751 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:38.318774 3751 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:38.319097 3751 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:38.319422 3751 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:38.319780 3751 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:38.320154 3751 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:38.322898 3751 tablet_loader.cc:96] loaded metadata for tablet 2804dc5bb8f940cca6515831b9b30c72 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:38.323382 3751 tablet_loader.cc:96] loaded metadata for tablet 56460f95360a413b8b3cb31195d4f102 (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:38.323674 3751 tablet_loader.cc:96] loaded metadata for tablet 7a4ef48a5e67406297cf21413236f563 (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:38.324285 3751 tablet_loader.cc:96] loaded metadata for tablet 95be079f58414b529a5f2009f56bb009 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:38.324746 3751 tablet_loader.cc:96] loaded metadata for tablet a6d924df2b9244feb4813bafa9a218b3 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:38.325182 3751 tablet_loader.cc:96] loaded metadata for tablet c022e7ee959d4d19958437a1ed855bbd (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:38.325500 3751 tablet_loader.cc:96] loaded metadata for tablet f08841e4d54446699cd387180cdd871a (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:38.325944 3751 tablet_loader.cc:96] loaded metadata for tablet f7fceab9655d4d9ea0e142f0e3a4d06e (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:38.326247 3751 tablet_loader.cc:96] loaded metadata for tablet ff5338af10954f3f9ee797c37a1ca9cc (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:38.326484 3751 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:38.327522 3751 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:38.327607 3751 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:38.329027 3751 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:38.329996 3751 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Loaded TSK: 0
I20250905 07:50:38.330854 3751 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:38.331763 3751 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:38.332713 3751 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:38.372690 3506 master_service.cc:948] ReplaceTablet: received request to replace tablet 7a4ef48a5e67406297cf21413236f563 from {username='slave'} at 127.0.0.1:57070
I20250905 07:50:38.373126 3504 master_service.cc:948] ReplaceTablet: received request to replace tablet 3743b39493ed452e89b1e3223bb1ba88 from {username='slave'} at 127.0.0.1:57070
I20250905 07:50:38.374634 3505 master_service.cc:948] ReplaceTablet: received request to replace tablet 56460f95360a413b8b3cb31195d4f102 from {username='slave'} at 127.0.0.1:57070
I20250905 07:50:38.374269 3498 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57070:
name: "default.Table_7c0359e055804d0b9a84bd4dd5e60634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:38.375963 3507 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a57e0ff6b6a422aa0e69678b7bd6762 from {username='slave'} at 127.0.0.1:57070
I20250905 07:50:38.376255 3503 master_service.cc:948] ReplaceTablet: received request to replace tablet 56460f95360a413b8b3cb31195d4f102 from {username='slave'} at 127.0.0.1:57070
I20250905 07:50:38.376466 3502 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57070:
table { table_name: "default.Table_9a0733038d2145b2a3ea38308d4b6fa8" } new_table_name: "default.Table_96eada578e0d465cb55ab0848c68acaa" modify_external_catalogs: true
I20250905 07:50:38.376726 3501 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57070:
table { table_name: "default.Table_df9d719332a94674bf8d0b4f1f9d6a7f" } new_table_name: "default.Table_30a6668a12364eedb06bd2b9bfbf4b31" modify_external_catalogs: true
I20250905 07:50:38.376917 3507 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57070:
name: "default.Table_e3daa55fc9e5447f82536064de82c58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:38.377496 3499 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57070:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:38.377869 3499 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57070:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:38.379762 3504 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57070:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:38.380069 3504 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57070:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:38.377163 3500 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57070:
name: "default.Table_347a8cac73e54038b34319425f8f3bd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:38.389775 3689 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Refusing update from remote peer 65033f3e4b44455389e29a5e1c9f82cd: Log matching property violated. Preceding OpId in replica: term: 7 index: 41. Preceding OpId from leader: term: 10 index: 52. (index mismatch)
I20250905 07:50:38.391438 3753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 40, Time since last communication: 0.000s
W20250905 07:50:38.393752 3493 consensus_peers.cc:489] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd -> Peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Couldn't send request to peer 81e2a8aedb144e61891de4dcd390ff35. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250905 07:50:38.396788 3559 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f9e3e731dcd google::LogMessage::Fail() at ??:0
@ 0x7f9e3e735b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9e3e7317cc google::LogMessage::Flush() at ??:0
@ 0x7f9e3e731a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9e54bad7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9e54bac061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f9e54bb3cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f9e54bb3b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250905 07:50:38.441742 3723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65033f3e4b44455389e29a5e1c9f82cd. Latest consensus state: current_term: 10 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:38.442029 3723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:38.442660 3752 mvcc.cc:204] Tried to move back new op lower bound from 7196912182813650944 to 7196912182528880640. Current Snapshot: MvccSnapshot[applied={T|T < 7196912182813650944}]
I20250905 07:50:38.442924 3722 mvcc.cc:204] Tried to move back new op lower bound from 7196912168518316032 to 7196912152968192000. Current Snapshot: MvccSnapshot[applied={T|T < 7196912168518316032}]
@ 0x7f9e4149a2b9 kudu::Thread::SuperviseThread() at ??:0
I20250905 07:50:38.445158 3723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:38.445400 3723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:38.443969 3763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:38.445839 3763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:38.446215 3759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "65033f3e4b44455389e29a5e1c9f82cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:38.446475 3759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9e448f16db start_thread at ??:0
I20250905 07:50:38.449154 3506 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Sending DeleteTablet for 3 replicas of tablet 7a4ef48a5e67406297cf21413236f563
@ 0x7f9e3d5e271f clone at ??:0
W20250905 07:50:38.645690 2673 connection.cc:537] client connection to 127.1.178.189:43391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:38.646410 2673 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.189:43391)
W20250905 07:50:38.646828 2673 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.189:43391)
W20250905 07:50:38.647225 2673 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.189:43391)
W20250905 07:50:38.647519 2673 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.189:43391)
I20250905 07:50:38.704702 3679 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:49927; Asking this server to re-register.
I20250905 07:50:38.704656 3677 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:53179; Asking this server to re-register.
I20250905 07:50:38.705572 2386 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:38.705616 2529 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:38.707341 3677 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:38.707464 3679 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:38.945154 3759 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:38.945367 3759 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:38.947270 3759 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:38.948176 3663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
W20250905 07:50:38.949374 3665 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:38.949592 3663 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:38.949725 3663 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:38.950022 3759 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250905 07:50:39.460544 3780 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:39.460763 3780 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
W20250905 07:50:39.465761 3663 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:39.465767 3780 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:39.466472 3665 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:39.466650 3665 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:39.466925 3780 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250905 07:50:40.529513 3795 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:40.529847 3795 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:40.530673 3795 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:40.532864 3665 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:40.533034 3663 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
I20250905 07:50:40.533136 3663 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:40.533349 3795 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250905 07:50:41.148584 3806 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:41.148799 3806 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:41.149446 3806 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:41.151386 3663 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391): Network error: Client connection negotiation failed: client connection to 127.1.178.189:43391: connect: Connection refused (error 111)
W20250905 07:50:41.151584 3665 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:41.151692 3665 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dbf15725a7342da830f57c0535eb72d; no voters: 65033f3e4b44455389e29a5e1c9f82cd, 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:41.151906 3806 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250905 07:50:41.513918 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--webserver_interface=127.1.178.189
--webserver_port=37293
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:41.857441 3816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:41.857851 3816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:41.857995 3816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:41.866513 3816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:41.866636 3816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:41.866753 3816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:41.866823 3816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:41.876982 3816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.178.189
--webserver_port=37293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:41.878721 3816 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:41.880614 3816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:41.889578 3826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:41.889762 3827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:41.891010 3829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:41.890971 3816 server_base.cc:1047] running on GCE node
I20250905 07:50:41.891860 3816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:41.893543 3816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:41.894769 3816 hybrid_clock.cc:648] HybridClock initialized: now 1757058641894709 us; error 51 us; skew 500 ppm
I20250905 07:50:41.895133 3816 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:41.897542 3816 webserver.cc:480] Webserver started at http://127.1.178.189:37293/ using document root <none> and password file <none>
I20250905 07:50:41.898339 3816 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:41.898466 3816 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:41.904843 3816 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250905 07:50:41.908234 3835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:41.909794 3816 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250905 07:50:41.909991 3816 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
uuid: "65033f3e4b44455389e29a5e1c9f82cd"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:41.910786 3816 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:41.929088 3816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:41.930002 3816 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:41.930291 3816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:41.951248 3816 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.189:43391
I20250905 07:50:41.951267 3886 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.189:43391 every 8 connection(s)
I20250905 07:50:41.953327 3816 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
I20250905 07:50:41.955556 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 3816
I20250905 07:50:41.967232 3887 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:41.972702 3887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap starting.
I20250905 07:50:41.986580 3887 log.cc:826] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:42.024909 3887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=51 ignored=0} inserts{seen=33 ignored=0} mutations{seen=37 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 07:50:42.025593 3887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap complete.
I20250905 07:50:42.031548 3887 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 10 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:42.034588 3887 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Initialized, Role: FOLLOWER
I20250905 07:50:42.035317 3887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 10.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:42.036217 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:42.036494 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:42.037348 3887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:42.054911 3908 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:42.055051 3908 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: loading cluster ID for follower catalog manager: success
I20250905 07:50:42.060371 3908 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: acquiring CA information for follower catalog manager: success
I20250905 07:50:42.063789 3908 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:42.084962 3816 master_runner.cc:186] Error getting master registration for 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111),
I20250905 07:50:42.089066 3816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:42.136698 3913 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:42.136922 3913 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:42.137596 3913 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:42.147467 3665 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:42.150972 3862 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 11 candidate_status { last_received { term: 10 index: 52 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:42.151535 3862 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 10.
I20250905 07:50:42.152134 3663 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:42.152474 3913 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250905 07:50:42.152609 3913 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 65033f3e4b44455389e29a5e1c9f82cd)
I20250905 07:50:42.152726 3913 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 10 FOLLOWER]: Advancing to term 11
I20250905 07:50:42.154845 3913 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:42.155488 3913 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 election: Requested vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:42.155973 3862 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 11 candidate_status { last_received { term: 10 index: 52 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:42.156195 3862 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 10 FOLLOWER]: Advancing to term 11
W20250905 07:50:42.157743 3665 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:42.160630 3862 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 11.
I20250905 07:50:42.161139 3663 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:42.161448 3913 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Leader election won for term 11
I20250905 07:50:42.161945 3913 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 LEADER]: Becoming Leader. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Running, Role: LEADER
I20250905 07:50:42.162460 3913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 10.52, Last appended by leader: 52, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:42.164317 3917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 11 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:42.164662 3917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:42.165431 3919 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:42.168190 3919 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:42.168643 3919 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:42.168996 3919 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:42.169350 3919 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:42.169675 3919 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:42.170018 3919 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:42.170339 3919 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:42.170687 3919 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:42.171016 3919 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:42.173568 3919 tablet_loader.cc:96] loaded metadata for tablet 193e1bbb2cf74b4885a3619c3d29899f (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:42.174036 3919 tablet_loader.cc:96] loaded metadata for tablet 2804dc5bb8f940cca6515831b9b30c72 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:42.175091 3919 tablet_loader.cc:96] loaded metadata for tablet 95be079f58414b529a5f2009f56bb009 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:42.175572 3919 tablet_loader.cc:96] loaded metadata for tablet a6d924df2b9244feb4813bafa9a218b3 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:42.176050 3919 tablet_loader.cc:96] loaded metadata for tablet b1e22c8f127247f0b26025c3d95937d3 (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:42.176357 3919 tablet_loader.cc:96] loaded metadata for tablet c022e7ee959d4d19958437a1ed855bbd (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:42.176748 3919 tablet_loader.cc:96] loaded metadata for tablet f08841e4d54446699cd387180cdd871a (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:42.177253 3919 tablet_loader.cc:96] loaded metadata for tablet f7fceab9655d4d9ea0e142f0e3a4d06e (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:42.177639 3919 tablet_loader.cc:96] loaded metadata for tablet ff5338af10954f3f9ee797c37a1ca9cc (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:42.177894 3919 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:42.179018 3919 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:42.179098 3919 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:42.180454 3919 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:42.181437 3919 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Loaded TSK: 0
I20250905 07:50:42.182435 3919 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:42.183844 3919 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:42.185242 3919 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:42.193176 2387 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:42.197029 3851 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:40187; Asking this server to re-register.
I20250905 07:50:42.197786 2387 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:42.200240 3851 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:42.201145 2530 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:42.202266 3851 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:47451; Asking this server to re-register.
I20250905 07:50:42.202773 2530 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:42.203888 3851 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:42.234445 3679 master_service.cc:948] ReplaceTablet: received request to replace tablet f08841e4d54446699cd387180cdd871a from {username='slave'} at 127.0.0.1:59458
I20250905 07:50:42.235052 3672 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59458:
table { table_name: "default.Table_146568e744294fdd8f082cdbd8451039" } new_table_name: "default.Table_bcc153425f724fa0aee1ba610df560ce" modify_external_catalogs: true
I20250905 07:50:42.236261 3671 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59458:
table { table_name: "default.Table_a908ec5ae1aa46e8b872ff681d610476" } new_table_name: "default.Table_23be441c886b4e17929155f056fdb23b" modify_external_catalogs: true
I20250905 07:50:42.236184 3673 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59458:
name: "default.Table_c6a86ab58c8f4eddbf3e0c4edf058247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:42.236949 3670 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59458:
name: "default.Table_6abbacd07c2f4ab4b74761cd100dcf58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:42.237188 3674 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59458:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:42.237495 3674 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59458:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:42.238106 3675 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59458:
name: "default.Table_4907c91834874a1a88a3312b6a0be96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:42.238879 3676 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59458:
name: "default.Table_cd8e6f84c6614cd58e55c59c90383205"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:42.239611 3678 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59458:
name: "default.Table_b3cf8c80578f4b4f84327aedafee6a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:42.241456 3677 master_service.cc:948] ReplaceTablet: received request to replace tablet 7a4ef48a5e67406297cf21413236f563 from {username='slave'} at 127.0.0.1:59458
I20250905 07:50:42.242522 3677 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59458:
table { table_name: "default.Table_9a0733038d2145b2a3ea38308d4b6fa8" } new_table_name: "default.Table_96eada578e0d465cb55ab0848c68acaa" modify_external_catalogs: true
I20250905 07:50:42.246615 3862 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Refusing update from remote peer 9dbf15725a7342da830f57c0535eb72d: Log matching property violated. Preceding OpId in replica: term: 10 index: 52. Preceding OpId from leader: term: 11 index: 54. (index mismatch)
W20250905 07:50:42.246784 3665 consensus_peers.cc:489] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d -> Peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Couldn't send request to peer 81e2a8aedb144e61891de4dcd390ff35. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:42.247500 3913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 51, Time since last communication: 0.000s
F20250905 07:50:42.248847 3736 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f98908addcd google::LogMessage::Fail() at ??:0
I20250905 07:50:42.252954 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 11 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:42.253298 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f98908b1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f98908ad7cc google::LogMessage::Flush() at ??:0
@ 0x7f98908ada11 google::LogMessage::~LogMessage() at ??:0
I20250905 07:50:42.258093 3893 mvcc.cc:204] Tried to move back new op lower bound from 7196912198631305216 to 7196912198301126656. Current Snapshot: MvccSnapshot[applied={T|T < 7196912198631305216}]
I20250905 07:50:42.258486 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:42.258710 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f98a6d297b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f98a6d28061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250905 07:50:42.291054 3917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:42.291296 2668 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:42.291339 3917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:42.292433 3679 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet f08841e4d54446699cd387180cdd871a
I20250905 07:50:42.293488 3851 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:52283; Asking this server to re-register.
I20250905 07:50:42.294199 2668 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:42.295679 3679 catalog_manager.cc:6652] ReplaceTablet: tablet f08841e4d54446699cd387180cdd871a deleted and replaced by tablet 691168915f6c40958ec129f76ba9fd1a
I20250905 07:50:42.295681 3851 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:42.297264 3679 master_service.cc:948] ReplaceTablet: received request to replace tablet 56460f95360a413b8b3cb31195d4f102 from {username='slave'} at 127.0.0.1:59458
I20250905 07:50:42.298132 3679 master_service.cc:948] ReplaceTablet: received request to replace tablet 56460f95360a413b8b3cb31195d4f102 from {username='slave'} at 127.0.0.1:59458
I20250905 07:50:42.298972 3679 master_service.cc:948] ReplaceTablet: received request to replace tablet 7a4ef48a5e67406297cf21413236f563 from {username='slave'} at 127.0.0.1:59458
I20250905 07:50:42.299877 3679 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59458:
table { table_name: "default.Table_df9d719332a94674bf8d0b4f1f9d6a7f" } new_table_name: "default.Table_30a6668a12364eedb06bd2b9bfbf4b31" modify_external_catalogs: true
@ 0x7f98a6d2fcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f98a6d2fb59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f98936162b9 kudu::Thread::SuperviseThread() at ??:0
I20250905 07:50:42.328127 2318 tablet_service.cc:1515] Processing DeleteTablet for tablet f08841e4d54446699cd387180cdd871a with delete_type TABLET_DATA_DELETED (replaced by tablet 691168915f6c40958ec129f76ba9fd1a) from {username='slave'} at 127.0.0.1:34258
I20250905 07:50:42.328146 2598 tablet_service.cc:1515] Processing DeleteTablet for tablet f08841e4d54446699cd387180cdd871a with delete_type TABLET_DATA_DELETED (replaced by tablet 691168915f6c40958ec129f76ba9fd1a) from {username='slave'} at 127.0.0.1:48596
I20250905 07:50:42.328567 2317 tablet_service.cc:1468] Processing CreateTablet for tablet 193e1bbb2cf74b4885a3619c3d29899f (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:42.329083 2317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193e1bbb2cf74b4885a3619c3d29899f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:42.337368 2320 tablet_service.cc:1468] Processing CreateTablet for tablet b1e22c8f127247f0b26025c3d95937d3 (DEFAULT_TABLE table=default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:42.337847 2320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e22c8f127247f0b26025c3d95937d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:42.340687 2601 tablet_service.cc:1468] Processing CreateTablet for tablet 193e1bbb2cf74b4885a3619c3d29899f (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:42.341112 2601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193e1bbb2cf74b4885a3619c3d29899f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:42.344405 3944 tablet_bootstrap.cc:492] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:42.346506 3944 tablet_bootstrap.cc:654] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
@ 0x7f9896a6d6db start_thread at ??:0
@ 0x7f988f75e71f clone at ??:0
I20250905 07:50:42.353518 3941 tablet_replica.cc:331] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:42.353842 3941 raft_consensus.cc:2241] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:42.354137 3941 raft_consensus.cc:2270] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:42.354984 3941 ts_tablet_manager.cc:1905] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:42.355701 3944 tablet_bootstrap.cc:492] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:42.355860 3944 ts_tablet_manager.cc:1397] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20250905 07:50:42.356834 3944 raft_consensus.cc:357] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:42.357065 3944 raft_consensus.cc:383] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:42.357167 3944 raft_consensus.cc:738] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:42.357491 3944 consensus_queue.cc:260] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:42.358259 2386 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:42.358342 3946 tablet_bootstrap.cc:492] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:42.360459 3942 tablet_replica.cc:331] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:42.360678 3942 raft_consensus.cc:2241] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:42.360459 3946 tablet_bootstrap.cc:654] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:42.360934 3942 raft_consensus.cc:2270] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:42.361735 3942 ts_tablet_manager.cc:1905] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:42.364915 3944 ts_tablet_manager.cc:1428] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250905 07:50:42.366557 3941 ts_tablet_manager.cc:1918] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:42.366705 3941 log.cc:1199] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/f08841e4d54446699cd387180cdd871a
I20250905 07:50:42.367115 3941 ts_tablet_manager.cc:1939] T f08841e4d54446699cd387180cdd871a P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:42.370991 3942 ts_tablet_manager.cc:1918] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:42.371172 3942 log.cc:1199] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/f08841e4d54446699cd387180cdd871a
I20250905 07:50:42.371773 3942 ts_tablet_manager.cc:1939] T f08841e4d54446699cd387180cdd871a P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:42.372589 3944 tablet_bootstrap.cc:492] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:42.373450 3946 tablet_bootstrap.cc:492] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:42.373593 3946 ts_tablet_manager.cc:1397] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20250905 07:50:42.374415 3944 tablet_bootstrap.cc:654] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:42.374526 3946 raft_consensus.cc:357] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:42.374733 3946 raft_consensus.cc:383] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:42.374830 3946 raft_consensus.cc:738] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:42.375077 3946 consensus_queue.cc:260] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:42.375763 2667 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:42.379680 3946 ts_tablet_manager.cc:1428] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250905 07:50:42.381171 3944 tablet_bootstrap.cc:492] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:42.381441 3944 ts_tablet_manager.cc:1397] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20250905 07:50:42.382263 3944 raft_consensus.cc:357] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:42.382558 3944 raft_consensus.cc:383] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:42.382714 3944 raft_consensus.cc:738] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:42.383065 3944 consensus_queue.cc:260] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:42.384613 3944 ts_tablet_manager.cc:1428] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250905 07:50:42.566046 2276 connection.cc:537] server connection from 127.0.0.1:34258 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:42.566255 2553 connection.cc:537] server connection from 127.0.0.1:48596 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:42.566363 3839 connection.cc:537] server connection from 127.0.0.1:57966 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:42.566716 2273 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:42.567266 2673 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:42.568769 2552 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:42.568809 2673 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.190:35357)
W20250905 07:50:42.569448 2673 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:42.569754 2673 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:42.569933 2673 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.190:35357)
I20250905 07:50:42.582352 3892 raft_consensus.cc:491] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:42.582542 3892 raft_consensus.cc:513] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:42.583225 3892 leader_election.cc:290] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:42.583688 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "193e1bbb2cf74b4885a3619c3d29899f" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:42.583942 2340 raft_consensus.cc:2466] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:42.584368 2552 leader_election.cc:304] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:42.584633 3892 raft_consensus.cc:2802] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:42.584754 3892 raft_consensus.cc:491] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:42.584870 3892 raft_consensus.cc:3058] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
W20250905 07:50:42.584764 2552 leader_election.cc:343] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: 193e1bbb2cf74b4885a3619c3d29899f
I20250905 07:50:42.587319 3892 raft_consensus.cc:513] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:42.587882 3892 leader_election.cc:290] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Requested vote from peers b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:42.588416 2340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "193e1bbb2cf74b4885a3619c3d29899f" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:42.588609 2340 raft_consensus.cc:3058] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
W20250905 07:50:42.588716 2552 leader_election.cc:343] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: 193e1bbb2cf74b4885a3619c3d29899f
I20250905 07:50:42.590463 2340 raft_consensus.cc:2466] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:42.590801 2552 leader_election.cc:304] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114
I20250905 07:50:42.591065 3892 raft_consensus.cc:2802] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:42.591601 3892 raft_consensus.cc:695] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Becoming Leader. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Running, Role: LEADER
I20250905 07:50:42.592517 3892 consensus_queue.cc:237] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:42.725381 2529 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
W20250905 07:50:42.800925 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:42.825752 3955 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:42.825966 3955 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:42.827761 3955 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:42.828675 3836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:42.829878 3838 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:42.830082 3836 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:42.830227 3836 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:42.830539 3955 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 07:50:42.875587 3938 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:42.875772 3938 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:42.876542 3938 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:50:42.877552 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:50:42.877776 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:50:42.877854 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:50:42.878080 3938 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:50:43.149770 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:43.399752 3892 consensus_queue.cc:1035] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 07:50:43.404564 2552 consensus_peers.cc:489] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 -> Peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Couldn't send request to peer 0f0103de40bc4724a20d7dc4bff02114. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 193e1bbb2cf74b4885a3619c3d29899f. This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:43.409952 3967 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:43.410105 3967 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:43.410701 3967 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:43.412293 3836 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:43.412436 3838 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:43.412617 3838 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:43.412829 3967 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 07:50:44.334070 3979 ts_tablet_manager.cc:927] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Initiating tablet copy from peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:44.335107 3979 tablet_copy_client.cc:323] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: tablet copy: Beginning tablet copy session from remote peer at address 127.1.178.131:45275
I20250905 07:50:44.336162 2641 tablet_copy_service.cc:140] P 715f96a2595d4c4e9eb0d68aea3465f4: Received BeginTabletCopySession request for tablet 193e1bbb2cf74b4885a3619c3d29899f from peer 0f0103de40bc4724a20d7dc4bff02114 ({username='slave'} at 127.1.178.130:50027)
I20250905 07:50:44.336355 2641 tablet_copy_service.cc:161] P 715f96a2595d4c4e9eb0d68aea3465f4: Beginning new tablet copy session on tablet 193e1bbb2cf74b4885a3619c3d29899f from peer 0f0103de40bc4724a20d7dc4bff02114 at {username='slave'} at 127.1.178.130:50027: session id = 0f0103de40bc4724a20d7dc4bff02114-193e1bbb2cf74b4885a3619c3d29899f
I20250905 07:50:44.338917 2641 tablet_copy_source_session.cc:215] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 07:50:44.341012 3979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193e1bbb2cf74b4885a3619c3d29899f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:44.346269 3979 tablet_copy_client.cc:806] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: tablet copy: Starting download of 0 data blocks...
I20250905 07:50:44.346554 3979 tablet_copy_client.cc:670] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: tablet copy: Starting download of 1 WAL segments...
I20250905 07:50:44.348913 3979 tablet_copy_client.cc:538] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 07:50:44.351666 3979 tablet_bootstrap.cc:492] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:44.357156 3979 tablet_bootstrap.cc:492] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:50:44.357548 3979 tablet_bootstrap.cc:492] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap complete.
I20250905 07:50:44.357771 3979 ts_tablet_manager.cc:1397] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20250905 07:50:44.358440 3979 raft_consensus.cc:357] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:44.358606 3979 raft_consensus.cc:738] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:44.358845 3979 consensus_queue.cc:260] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:44.359346 3979 ts_tablet_manager.cc:1428] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 07:50:44.360710 2641 tablet_copy_service.cc:342] P 715f96a2595d4c4e9eb0d68aea3465f4: Request end of tablet copy session 0f0103de40bc4724a20d7dc4bff02114-193e1bbb2cf74b4885a3619c3d29899f received from {username='slave'} at 127.1.178.130:50027
I20250905 07:50:44.361200 2641 tablet_copy_service.cc:434] P 715f96a2595d4c4e9eb0d68aea3465f4: ending tablet copy session 0f0103de40bc4724a20d7dc4bff02114-193e1bbb2cf74b4885a3619c3d29899f on tablet 193e1bbb2cf74b4885a3619c3d29899f with peer 0f0103de40bc4724a20d7dc4bff02114
I20250905 07:50:44.473574 1738 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu pid 3816
I20250905 07:50:44.477149 3987 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:44.477386 3987 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:44.478053 3987 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:44.480614 3838 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:44.480893 3836 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:44.480995 3836 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:44.481225 3987 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 07:50:44.485450 3882 generic_service.cc:196] Checking for leaks (request via RPC)
I20250905 07:50:44.594944 1738 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu with pid 3816
I20250905 07:50:45.549475 2483 raft_consensus.cc:1215] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250905 07:50:45.594475 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--webserver_interface=127.1.178.189
--webserver_port=37293
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:45.940634 4012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:45.940994 4012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:45.941121 4012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:45.949409 4012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:45.949532 4012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:45.949616 4012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:45.949678 4012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:45.962232 4012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.189:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.178.189
--webserver_port=37293
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:45.964380 4012 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:45.966815 4012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:45.976235 4018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:45.976253 4019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:45.977372 4021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:45.978037 4012 server_base.cc:1047] running on GCE node
I20250905 07:50:45.978811 4012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:45.980288 4012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:45.981494 4012 hybrid_clock.cc:648] HybridClock initialized: now 1757058645981428 us; error 51 us; skew 500 ppm
I20250905 07:50:45.981844 4012 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:45.984133 4012 webserver.cc:480] Webserver started at http://127.1.178.189:37293/ using document root <none> and password file <none>
I20250905 07:50:45.984930 4012 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:45.985040 4012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:45.990608 4012 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 07:50:45.993673 4027 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:45.995201 4012 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250905 07:50:45.995370 4012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
uuid: "65033f3e4b44455389e29a5e1c9f82cd"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:45.996163 4012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:46.014463 4012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:46.015413 4012 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:46.015710 4012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:46.035473 4012 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.189:43391
I20250905 07:50:46.035494 4078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.189:43391 every 8 connection(s)
I20250905 07:50:46.037308 4012 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-1/data/info.pb
I20250905 07:50:46.044423 4079 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:46.045337 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 4012
I20250905 07:50:46.048708 4079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap starting.
I20250905 07:50:46.080037 2668 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:46.084044 4079 log.cc:826] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:46.085530 2530 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:46.129979 4079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=55 ignored=0} inserts{seen=35 ignored=0} mutations{seen=41 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 07:50:46.130582 4079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: Bootstrap complete.
I20250905 07:50:46.135752 4079 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:46.136492 4079 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65033f3e4b44455389e29a5e1c9f82cd, State: Initialized, Role: FOLLOWER
I20250905 07:50:46.137146 4079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 55, Last appended: 11.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:46.137976 4092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 11 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:46.138212 4092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:46.139031 4079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:46.150013 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:50:46.154224 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:46.154292 4106 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:46.154532 4106 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: loading cluster ID for follower catalog manager: success
I20250905 07:50:46.162036 4106 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: acquiring CA information for follower catalog manager: success
I20250905 07:50:46.165566 4106 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:46.359387 4110 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:46.359576 4110 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:46.360296 4110 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:50:46.361460 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:50:46.361702 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:50:46.361826 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:50:46.362169 4110 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:50:46.421663 2387 heartbeater.cc:344] Connected to a master server at 127.1.178.189:43391
I20250905 07:50:46.423853 4044 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:43591; Asking this server to re-register.
I20250905 07:50:46.424571 2387 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:46.426376 4044 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:46.518185 4092 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:46.518497 4092 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:46.520833 4092 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:46.521806 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:46.523301 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:46.523545 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:46.523734 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:46.524149 4092 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 07:50:46.586449 4044 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:59141; Asking this server to re-register.
I20250905 07:50:46.587117 2668 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:46.588459 4044 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:46.590837 4044 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:42247; Asking this server to re-register.
I20250905 07:50:46.591419 2530 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:46.592607 4044 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:47.248457 4122 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:47.248669 4122 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:47.249472 4122 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:47.251586 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:47.251771 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:47.251963 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:47.252220 4122 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 07:50:47.940310 4132 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:47.940541 4132 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:47.941231 4132 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:47.943269 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:47.943444 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:47.943614 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:47.943871 4132 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 07:50:49.089583 1738 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
/tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--webserver_interface=127.1.178.190
--webserver_port=34789
--builtin_ntp_servers=127.1.178.148:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 07:50:49.429292 4149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 07:50:49.429651 4149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 07:50:49.429796 4149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 07:50:49.438038 4149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 07:50:49.438158 4149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 07:50:49.438280 4149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 07:50:49.438361 4149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 07:50:49.448983 4149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.178.148:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45227
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.178.190:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.178.190
--webserver_port=34789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 07:43:20 UTC on 24a791456cd2
build id 7908
ASAN enabled
I20250905 07:50:49.450649 4149 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 07:50:49.452621 4149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 07:50:49.461627 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 07:50:49.461750 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:49.462170 4162 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:49.462532 4162 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:49.478304 4162 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:49.489233 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:49.489962 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:49.490123 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
W20250905 07:50:49.490458 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 07:50:49.490541 4162 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 07:50:49.491009 4149 server_base.cc:1047] running on GCE node
I20250905 07:50:49.491909 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 07:50:49.493597 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 07:50:49.494823 4149 hybrid_clock.cc:648] HybridClock initialized: now 1757058649494774 us; error 44 us; skew 500 ppm
I20250905 07:50:49.495262 4149 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 07:50:49.497812 4149 webserver.cc:480] Webserver started at http://127.1.178.190:34789/ using document root <none> and password file <none>
I20250905 07:50:49.498703 4149 fs_manager.cc:362] Metadata directory not provided
I20250905 07:50:49.498827 4149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 07:50:49.505048 4149 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250905 07:50:49.508425 4172 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 07:50:49.509944 4149 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250905 07:50:49.510147 4149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data,/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
uuid: "9dbf15725a7342da830f57c0535eb72d"
format_stamp: "Formatted at 2025-09-05 07:50:15 on dist-test-slave-tgv6"
I20250905 07:50:49.510969 4149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 07:50:49.527444 4149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 07:50:49.528410 4149 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 07:50:49.528730 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 07:50:49.550189 4149 rpc_server.cc:307] RPC server started. Bound to: 127.1.178.190:35357
I20250905 07:50:49.550204 4223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.178.190:35357 every 8 connection(s)
I20250905 07:50:49.552079 4149 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/master-0/data/info.pb
I20250905 07:50:49.556558 1738 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7ZTeCZ/build/asan/bin/kudu as pid 4149
I20250905 07:50:49.559988 4224 sys_catalog.cc:263] Verifying existing consensus state
I20250905 07:50:49.565717 4224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap starting.
I20250905 07:50:49.582208 2667 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:49.582340 4224 log.cc:826] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Log is configured to *not* fsync() on all Append() calls
I20250905 07:50:49.582463 2667 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:49.609582 2529 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:49.609786 2529 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:49.625730 4224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=55 ignored=0} inserts{seen=35 ignored=0} mutations{seen=41 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 07:50:49.626441 4224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Bootstrap complete.
I20250905 07:50:49.632213 4224 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:49.634742 4224 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Initialized, Role: FOLLOWER
I20250905 07:50:49.635373 4224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 55, Last appended: 11.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:49.636112 4230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 11 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:49.636360 4230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:49.637144 4224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 07:50:49.650316 4244 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:49.650426 4244 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: loading cluster ID for follower catalog manager: success
I20250905 07:50:49.655025 4244 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: acquiring CA information for follower catalog manager: success
I20250905 07:50:49.658485 4244 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 07:50:49.676458 4149 master_runner.cc:186] Error getting master registration for 127.1.178.188:39743: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111),
I20250905 07:50:49.680634 4149 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:49.933482 2386 heartbeater.cc:344] Connected to a master server at 127.1.178.190:35357
I20250905 07:50:49.933712 2386 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:49.936589 4189 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" instance_seqno: 1757058617448935) as {username='slave'} at 127.1.178.129:37779; Asking this server to re-register.
I20250905 07:50:49.937472 2386 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:49.940059 4189 ts_manager.cc:194] Registered new tserver with Master: b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:50.085938 2667 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:50.087873 4189 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" instance_seqno: 1757058618409032) as {username='slave'} at 127.1.178.131:34421; Asking this server to re-register.
I20250905 07:50:50.088572 2667 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:50.089744 4189 ts_manager.cc:194] Registered new tserver with Master: 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275)
I20250905 07:50:50.112269 2529 heartbeater.cc:499] Master 127.1.178.190:35357 was elected leader, sending a full tablet report...
I20250905 07:50:50.114229 4189 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" instance_seqno: 1757058617946299) as {username='slave'} at 127.1.178.130:54511; Asking this server to re-register.
I20250905 07:50:50.114955 2529 heartbeater.cc:461] Registering TS with master...
I20250905 07:50:50.116482 4189 ts_manager.cc:194] Registered new tserver with Master: 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
I20250905 07:50:50.240203 4251 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:50.240496 4251 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:50.242327 4251 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:50.242998 4054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 12 candidate_status { last_received { term: 11 index: 56 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd" is_pre_election: true
I20250905 07:50:50.243366 4054 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 11.
W20250905 07:50:50.243333 4175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:50.244052 4173 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters:
I20250905 07:50:50.244438 4251 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250905 07:50:50.244601 4251 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:50.244719 4251 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 11 FOLLOWER]: Advancing to term 12
W20250905 07:50:50.244742 4175 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:50.248299 4251 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:50.248824 4251 leader_election.cc:290] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 12 election: Requested vote from peers 65033f3e4b44455389e29a5e1c9f82cd (127.1.178.189:43391), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
I20250905 07:50:50.249230 4054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbf15725a7342da830f57c0535eb72d" candidate_term: 12 candidate_status { last_received { term: 11 index: 56 } } ignore_live_leader: false dest_uuid: "65033f3e4b44455389e29a5e1c9f82cd"
I20250905 07:50:50.249444 4054 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 11 FOLLOWER]: Advancing to term 12
W20250905 07:50:50.250083 4175 leader_election.cc:336] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:50.252882 4054 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbf15725a7342da830f57c0535eb72d in term 12.
I20250905 07:50:50.253245 4173 leader_election.cc:304] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd, 9dbf15725a7342da830f57c0535eb72d; no voters: 81e2a8aedb144e61891de4dcd390ff35
I20250905 07:50:50.253517 4251 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 12 FOLLOWER]: Leader election won for term 12
I20250905 07:50:50.253855 4251 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [term 12 LEADER]: Becoming Leader. State: Replica: 9dbf15725a7342da830f57c0535eb72d, State: Running, Role: LEADER
I20250905 07:50:50.254253 4251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 11.56, Last appended by leader: 56, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:50.256736 4254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 12 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:50.256994 4254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
I20250905 07:50:50.732254 4054 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Refusing update from remote peer 9dbf15725a7342da830f57c0535eb72d: Log matching property violated. Preceding OpId in replica: term: 11 index: 55. Preceding OpId from leader: term: 12 index: 57. (index mismatch)
I20250905 07:50:50.733392 4254 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 55, Time since last communication: 0.000s
I20250905 07:50:50.741449 4262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbf15725a7342da830f57c0535eb72d. Latest consensus state: current_term: 12 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:50.741690 4262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:50.744699 4262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:50.744985 4262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [sys.catalog]: This master's current role is: FOLLOWER
I20250905 07:50:50.747014 4254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "9dbf15725a7342da830f57c0535eb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } } }
I20250905 07:50:50.747225 4254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d [sys.catalog]: This master's current role is: LEADER
W20250905 07:50:50.748349 4175 consensus_peers.cc:489] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d -> Peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Couldn't send request to peer 81e2a8aedb144e61891de4dcd390ff35. Status: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:50.774237 4256 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 07:50:50.777217 4256 catalog_manager.cc:671] Loaded metadata for table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938]
I20250905 07:50:50.777688 4256 catalog_manager.cc:671] Loaded metadata for table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421]
I20250905 07:50:50.778046 4256 catalog_manager.cc:671] Loaded metadata for table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]
I20250905 07:50:50.778362 4256 catalog_manager.cc:671] Loaded metadata for table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550]
I20250905 07:50:50.778725 4256 catalog_manager.cc:671] Loaded metadata for table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a]
I20250905 07:50:50.779060 4256 catalog_manager.cc:671] Loaded metadata for table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7]
I20250905 07:50:50.779361 4256 catalog_manager.cc:671] Loaded metadata for table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]
I20250905 07:50:50.779686 4256 catalog_manager.cc:671] Loaded metadata for table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4]
I20250905 07:50:50.780056 4256 catalog_manager.cc:671] Loaded metadata for table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c]
I20250905 07:50:50.782493 4256 tablet_loader.cc:96] loaded metadata for tablet 193e1bbb2cf74b4885a3619c3d29899f (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688])
I20250905 07:50:50.782956 4256 tablet_loader.cc:96] loaded metadata for tablet 2804dc5bb8f940cca6515831b9b30c72 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb])
I20250905 07:50:50.783543 4256 tablet_loader.cc:96] loaded metadata for tablet 691168915f6c40958ec129f76ba9fd1a (table default.table_45c1508d77dc4693960b671c9894a5b2 [id=083a9a6a14344415ab3dbcc3b5095421])
I20250905 07:50:50.784261 4256 tablet_loader.cc:96] loaded metadata for tablet 95be079f58414b529a5f2009f56bb009 (table default.table_f040e3a613e6480bb336e73085713781 [id=fd61a1e7fe43485f9fc615f7fbe7320c])
I20250905 07:50:50.784677 4256 tablet_loader.cc:96] loaded metadata for tablet a6d924df2b9244feb4813bafa9a218b3 (table default.table_df9d719332a94674bf8d0b4f1f9d6a7f [id=07dcc92b52df4317b60c8f9acf93d938])
I20250905 07:50:50.785135 4256 tablet_loader.cc:96] loaded metadata for tablet b1e22c8f127247f0b26025c3d95937d3 (table default.table_a908ec5ae1aa46e8b872ff681d610476 [id=440fb67610434736a18cb4dc5948b87a])
I20250905 07:50:50.785439 4256 tablet_loader.cc:96] loaded metadata for tablet c022e7ee959d4d19958437a1ed855bbd (table default.table_9a0733038d2145b2a3ea38308d4b6fa8 [id=3022d080c03b49a6acdec8ceb0c36550])
I20250905 07:50:50.785991 4256 tablet_loader.cc:96] loaded metadata for tablet f7fceab9655d4d9ea0e142f0e3a4d06e (table default.table_fbd6c4d89fd54fdd9765576cc3cf5cbc [id=e5b95da4c2994c7b98ed5125803bf6e4])
I20250905 07:50:50.786268 4256 tablet_loader.cc:96] loaded metadata for tablet ff5338af10954f3f9ee797c37a1ca9cc (table default.table_146568e744294fdd8f082cdbd8451039 [id=7b9d729d8e7746ec8a605ec72650d4a7])
I20250905 07:50:50.786505 4256 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 07:50:50.788508 4256 catalog_manager.cc:1261] Loaded cluster ID: 104584ca7f7f4b0b87c7b98ecac553e8
I20250905 07:50:50.788596 4256 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 07:50:50.789953 4256 catalog_manager.cc:1506] Loading token signing keys...
I20250905 07:50:50.790933 4256 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Loaded TSK: 0
I20250905 07:50:50.791805 4256 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 07:50:50.792809 4256 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250905 07:50:50.793757 4256 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250905 07:50:50.803557 4189 master_service.cc:948] ReplaceTablet: received request to replace tablet 193e1bbb2cf74b4885a3619c3d29899f from {username='slave'} at 127.0.0.1:35504
I20250905 07:50:50.803607 4188 master_service.cc:948] ReplaceTablet: received request to replace tablet 083c74057a6945288b259678e0ff2223 from {username='slave'} at 127.0.0.1:35504
I20250905 07:50:50.803869 4186 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a57e0ff6b6a422aa0e69678b7bd6762 from {username='slave'} at 127.0.0.1:35504
I20250905 07:50:50.804111 4187 master_service.cc:948] ReplaceTablet: received request to replace tablet 2804dc5bb8f940cca6515831b9b30c72 from {username='slave'} at 127.0.0.1:35504
I20250905 07:50:50.805249 4185 master_service.cc:948] ReplaceTablet: received request to replace tablet 56460f95360a413b8b3cb31195d4f102 from {username='slave'} at 127.0.0.1:35504
I20250905 07:50:50.806198 4188 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35504:
name: "default.Table_e3daa55fc9e5447f82536064de82c58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:50.806021 4184 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35504:
name: "default.Table_06530af7099e45c7b150408f8c1a3436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:50.807790 4181 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
I20250905 07:50:50.805619 4180 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:50.807945 4185 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:50.808245 4186 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35504:
name: "default.Table_7c0359e055804d0b9a84bd4dd5e60634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 07:50:50.808975 4181 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_18a839c38dff41c892ed79bdaf6c8229" } modify_external_catalogs: true
I20250905 07:50:50.809188 4180 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7" } modify_external_catalogs: true
I20250905 07:50:50.808568 4182 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_fbd6c4d89fd54fdd9765576cc3cf5cbc" } new_table_name: "default.Table_68b9ab7fcb3245ad84e0fe0e5c05d746" modify_external_catalogs: true
I20250905 07:50:50.809660 4185 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_f040e3a613e6480bb336e73085713781" } modify_external_catalogs: true
I20250905 07:50:50.809015 4183 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35504:
name: "default.Table_8f62349a373e45b8ae063dafbfc3cdcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250905 07:50:50.816540 4243 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_cc7e423f936848e1b8abbb22c3440719, message={"server":"","servicePrincipal":"","db":"default","table":"table_cc7e423f936848e1b8abbb22c3440719","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_cc7e423f936848e1b8abbb22c3440719\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_cc7e423f936848e1b8abbb22c3440719\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_cc7e423f936848e1b8abbb22c3440719\",\"kudu.table_id\":\"52d2fe862d134f2eaaa2d5c719594e27\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1757058618, eventType=CREATE_TABLE, dbName=default, tableName=table_df9d719332a94674bf8d0b4f1f9d6a7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_df9d719332a94674bf8d0b4f1f9d6a7f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1757058618},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/warehouse/table_df9d719332a94674bf8d0b4f1f9d6a7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df9d719332a94674bf8d0b4f1f9d6a7f\",\"kudu.table_id\":\"07dcc92b52df4317b60c8f9acf93d938\",\"transient_lastDdlTime\":\"1757058618\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"104584ca7f7f4b0b87c7b98ecac553e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1757058618,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250905 07:50:50.819831 4187 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet 2804dc5bb8f940cca6515831b9b30c72
@ 0x7ff759eebdcd google::LogMessage::Fail() at ??:0
@ 0x7ff759eefb93 google::LogMessage::SendToLog() at ??:0
I20250905 07:50:50.822727 4187 catalog_manager.cc:6652] ReplaceTablet: tablet 2804dc5bb8f940cca6515831b9b30c72 deleted and replaced by tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43
I20250905 07:50:50.824391 4189 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9dbf15725a7342da830f57c0535eb72d: Sending DeleteTablet for 3 replicas of tablet 193e1bbb2cf74b4885a3619c3d29899f
I20250905 07:50:50.824990 4189 catalog_manager.cc:6652] ReplaceTablet: tablet 193e1bbb2cf74b4885a3619c3d29899f deleted and replaced by tablet badb79f796d240029f0d8bc90deaa5c6
@ 0x7ff759eeb7cc google::LogMessage::Flush() at ??:0
@ 0x7ff759eeba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ff7703677b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7ff770366061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250905 07:50:50.864801 4187 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:50.865095 4187 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35504:
table { table_name: "default.Table_45c1508d77dc4693960b671c9894a5b2" } modify_external_catalogs: true
I20250905 07:50:50.865137 2598 tablet_service.cc:1515] Processing DeleteTablet for tablet 2804dc5bb8f940cca6515831b9b30c72 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43) from {username='slave'} at 127.0.0.1:60258
I20250905 07:50:50.865151 2318 tablet_service.cc:1515] Processing DeleteTablet for tablet 2804dc5bb8f940cca6515831b9b30c72 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43) from {username='slave'} at 127.0.0.1:40112
I20250905 07:50:50.865418 2317 tablet_service.cc:1515] Processing DeleteTablet for tablet 193e1bbb2cf74b4885a3619c3d29899f with delete_type TABLET_DATA_DELETED (replaced by tablet badb79f796d240029f0d8bc90deaa5c6) from {username='slave'} at 127.0.0.1:40112
I20250905 07:50:50.865487 2601 tablet_service.cc:1515] Processing DeleteTablet for tablet 193e1bbb2cf74b4885a3619c3d29899f with delete_type TABLET_DATA_DELETED (replaced by tablet badb79f796d240029f0d8bc90deaa5c6) from {username='slave'} at 127.0.0.1:60258
I20250905 07:50:50.865990 2463 tablet_service.cc:1515] Processing DeleteTablet for tablet 2804dc5bb8f940cca6515831b9b30c72 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43) from {username='slave'} at 127.0.0.1:36602
I20250905 07:50:50.865983 2460 tablet_service.cc:1515] Processing DeleteTablet for tablet 193e1bbb2cf74b4885a3619c3d29899f with delete_type TABLET_DATA_DELETED (replaced by tablet badb79f796d240029f0d8bc90deaa5c6) from {username='slave'} at 127.0.0.1:36602
I20250905 07:50:50.868266 4189 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35504:
name: "default.Table_4907c91834874a1a88a3312b6a0be96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7ff77036dcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250905 07:50:50.873765 4283 tablet_replica.cc:331] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:50.874046 4283 raft_consensus.cc:2241] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:50.874358 4283 raft_consensus.cc:2270] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7ff77036db59 std::_Function_handler<>::_M_invoke() at ??:0
I20250905 07:50:50.875142 4283 ts_tablet_manager.cc:1905] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:50.876401 4282 tablet_replica.cc:331] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:50.876603 4282 raft_consensus.cc:2241] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:50.876811 4282 raft_consensus.cc:2270] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:50.877383 4282 ts_tablet_manager.cc:1905] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:50.877887 4281 tablet_replica.cc:331] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:50.878098 4281 raft_consensus.cc:2241] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:50:50.878485 4281 raft_consensus.cc:2270] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:50.879515 2601 tablet_service.cc:1468] Processing CreateTablet for tablet badb79f796d240029f0d8bc90deaa5c6 (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:50.879791 2460 tablet_service.cc:1468] Processing CreateTablet for tablet badb79f796d240029f0d8bc90deaa5c6 (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:50.879941 2601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet badb79f796d240029f0d8bc90deaa5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:50.880208 2460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet badb79f796d240029f0d8bc90deaa5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:50.881724 2317 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:50.881908 2463 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0xa774dd std::function<>::operator()() at ??:?
I20250905 07:50:50.882091 2317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:50.882364 2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:50.884485 2599 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43 (DEFAULT_TABLE table=default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:50.884902 2599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1fb1dd3cb94ca09890a94f8f1d0f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 07:50:50.888302 2318 tablet_service.cc:1468] Processing CreateTablet for tablet badb79f796d240029f0d8bc90deaa5c6 (DEFAULT_TABLE table=default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 07:50:50.890014 4281 ts_tablet_manager.cc:1905] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 07:50:50.891077 2552 consensus_peers.cc:489] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 -> Peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Couldn't send request to peer 0f0103de40bc4724a20d7dc4bff02114. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250905 07:50:50.892287 2318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet badb79f796d240029f0d8bc90deaa5c6. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ff75cc542b9 kudu::Thread::SuperviseThread() at ??:0
I20250905 07:50:50.895388 4285 tablet_bootstrap.cc:492] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
@ 0x7ff7600ab6db start_thread at ??:0
I20250905 07:50:50.897517 4285 tablet_bootstrap.cc:654] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:50.898135 4283 ts_tablet_manager.cc:1918] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:50.898269 4283 log.cc:1199] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/193e1bbb2cf74b4885a3619c3d29899f
I20250905 07:50:50.899055 4283 ts_tablet_manager.cc:1939] T 193e1bbb2cf74b4885a3619c3d29899f P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:50.900489 4173 catalog_manager.cc:4928] TS 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): tablet 193e1bbb2cf74b4885a3619c3d29899f (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]) successfully deleted
I20250905 07:50:50.900601 4283 tablet_replica.cc:331] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: stopping tablet replica
I20250905 07:50:50.900965 4282 ts_tablet_manager.cc:1918] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:50.901008 4283 raft_consensus.cc:2241] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:50.901095 4282 log.cc:1199] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/2804dc5bb8f940cca6515831b9b30c72
I20250905 07:50:50.901299 4283 raft_consensus.cc:2270] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:50.901580 4282 ts_tablet_manager.cc:1939] T 2804dc5bb8f940cca6515831b9b30c72 P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:50.902128 4283 ts_tablet_manager.cc:1905] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:50.902671 4282 tablet_replica.cc:331] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: stopping tablet replica
I20250905 07:50:50.902761 4173 catalog_manager.cc:4928] TS b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): tablet 2804dc5bb8f940cca6515831b9b30c72 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]) successfully deleted
I20250905 07:50:50.902882 4282 raft_consensus.cc:2241] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 07:50:50.903064 4282 raft_consensus.cc:2270] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:50.903614 4282 ts_tablet_manager.cc:1905] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:50.908804 4285 tablet_bootstrap.cc:492] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:50.908982 4285 ts_tablet_manager.cc:1397] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250905 07:50:50.909301 4283 ts_tablet_manager.cc:1918] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:50.909446 4283 log.cc:1199] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-1/wal/wals/2804dc5bb8f940cca6515831b9b30c72
I20250905 07:50:50.909967 4283 ts_tablet_manager.cc:1939] T 2804dc5bb8f940cca6515831b9b30c72 P 0f0103de40bc4724a20d7dc4bff02114: Deleting consensus metadata
I20250905 07:50:50.910036 4285 raft_consensus.cc:357] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:50.910283 4285 raft_consensus.cc:383] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:50.910432 4285 raft_consensus.cc:738] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:50.910836 4284 tablet_bootstrap.cc:492] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:50.911382 4282 ts_tablet_manager.cc:1918] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:50.911408 4281 ts_tablet_manager.cc:1918] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:50.911506 4282 log.cc:1199] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-0/wal/wals/193e1bbb2cf74b4885a3619c3d29899f
I20250905 07:50:50.911545 4281 log.cc:1199] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/2804dc5bb8f940cca6515831b9b30c72
I20250905 07:50:50.911986 4282 ts_tablet_manager.cc:1939] T 193e1bbb2cf74b4885a3619c3d29899f P b902ef48db944a02813b8543b2f91a7e: Deleting consensus metadata
I20250905 07:50:50.911952 4173 catalog_manager.cc:4928] TS 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): tablet 2804dc5bb8f940cca6515831b9b30c72 (table default.table_0ccc7b1d1bc24fd5a2485bc8b7c1a3f7 [id=1e1a843a866f421fb6deae9c8300d5bb]) successfully deleted
I20250905 07:50:50.912089 4281 ts_tablet_manager.cc:1939] T 2804dc5bb8f940cca6515831b9b30c72 P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:50.912410 4286 tablet_bootstrap.cc:492] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
@ 0x7ff758d9c71f clone at ??:0
I20250905 07:50:50.912901 4173 catalog_manager.cc:4928] TS b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): tablet 193e1bbb2cf74b4885a3619c3d29899f (table default.table_18a839c38dff41c892ed79bdaf6c8229 [id=84dae6b410994fb5ba4564a6bc3ba688]) successfully deleted
I20250905 07:50:50.913018 4284 tablet_bootstrap.cc:654] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:50.913457 4285 consensus_queue.cc:260] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:50.914058 4281 tablet_replica.cc:331] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: stopping tablet replica
I20250905 07:50:50.914280 4281 raft_consensus.cc:2241] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Raft consensus shutting down.
I20250905 07:50:50.914774 4281 raft_consensus.cc:2270] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 07:50:50.914978 4286 tablet_bootstrap.cc:654] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:50.915408 4281 ts_tablet_manager.cc:1905] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 07:50:50.920509 4285 ts_tablet_manager.cc:1428] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250905 07:50:50.925127 4285 tablet_bootstrap.cc:492] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e: Bootstrap starting.
I20250905 07:50:50.927502 4285 tablet_bootstrap.cc:654] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:50.932454 4281 ts_tablet_manager.cc:1918] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 07:50:50.932648 4281 log.cc:1199] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting WAL directory at /tmp/dist-test-task7ZTeCZ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1757058598803567-1738-0/minicluster-data/ts-2/wal/wals/193e1bbb2cf74b4885a3619c3d29899f
I20250905 07:50:50.933138 4281 ts_tablet_manager.cc:1939] T 193e1bbb2cf74b4885a3619c3d29899f P 715f96a2595d4c4e9eb0d68aea3465f4: Deleting consensus metadata
I20250905 07:50:50.935343 4284 tablet_bootstrap.cc:492] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:50.935389 4285 tablet_bootstrap.cc:492] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e: No bootstrap required, opened a new log
I20250905 07:50:50.935508 4284 ts_tablet_manager.cc:1397] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20250905 07:50:50.935542 4286 tablet_bootstrap.cc:492] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:50.935582 4285 ts_tablet_manager.cc:1397] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250905 07:50:50.935688 4286 ts_tablet_manager.cc:1397] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20250905 07:50:50.936357 4284 raft_consensus.cc:357] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.936368 4285 raft_consensus.cc:357] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.936553 4284 raft_consensus.cc:383] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:50.936483 4286 raft_consensus.cc:357] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:50.936589 4285 raft_consensus.cc:383] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:50.936656 4284 raft_consensus.cc:738] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:50.936663 4286 raft_consensus.cc:383] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:50.936681 4285 raft_consensus.cc:738] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902ef48db944a02813b8543b2f91a7e, State: Initialized, Role: FOLLOWER
I20250905 07:50:50.936748 4286 raft_consensus.cc:738] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:50.936878 4284 consensus_queue.cc:260] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.936957 4286 consensus_queue.cc:260] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:50.936997 4285 consensus_queue.cc:260] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.937537 4286 ts_tablet_manager.cc:1428] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 07:50:50.937796 4286 tablet_bootstrap.cc:492] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114: Bootstrap starting.
I20250905 07:50:50.939586 4286 tablet_bootstrap.cc:654] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:50.941891 4284 ts_tablet_manager.cc:1428] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 07:50:50.942162 4284 tablet_bootstrap.cc:492] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4: Bootstrap starting.
I20250905 07:50:50.943904 4284 tablet_bootstrap.cc:654] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4: Neither blocks nor log segments found. Creating new log.
I20250905 07:50:50.948302 4284 tablet_bootstrap.cc:492] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4: No bootstrap required, opened a new log
I20250905 07:50:50.948459 4284 ts_tablet_manager.cc:1397] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250905 07:50:50.949713 4285 ts_tablet_manager.cc:1428] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250905 07:50:50.950712 4284 raft_consensus.cc:357] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:50.950914 4284 raft_consensus.cc:383] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:50.951004 4284 raft_consensus.cc:738] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Initialized, Role: FOLLOWER
I20250905 07:50:50.951315 4284 consensus_queue.cc:260] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:50.954769 4284 ts_tablet_manager.cc:1428] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 07:50:50.961833 4286 tablet_bootstrap.cc:492] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114: No bootstrap required, opened a new log
I20250905 07:50:50.962018 4286 ts_tablet_manager.cc:1397] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.012s
I20250905 07:50:50.962947 4286 raft_consensus.cc:357] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.963160 4286 raft_consensus.cc:383] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 07:50:50.963250 4286 raft_consensus.cc:738] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Initialized, Role: FOLLOWER
I20250905 07:50:50.963505 4286 consensus_queue.cc:260] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.964092 4286 ts_tablet_manager.cc:1428] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250905 07:50:50.985589 4010 raft_consensus.cc:491] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:50.985765 4010 raft_consensus.cc:513] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.986783 4010 leader_election.cc:290] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:50.987393 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "badb79f796d240029f0d8bc90deaa5c6" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114" is_pre_election: true
I20250905 07:50:50.987447 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "badb79f796d240029f0d8bc90deaa5c6" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:50.987618 2483 raft_consensus.cc:2466] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:50.987637 2339 raft_consensus.cc:2466] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 0.
I20250905 07:50:50.988080 2552 leader_election.cc:304] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:50.988336 4010 raft_consensus.cc:2802] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:50.988449 4010 raft_consensus.cc:491] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:50.988543 4010 raft_consensus.cc:3058] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:50.990322 4010 raft_consensus.cc:513] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:50.990829 4010 leader_election.cc:290] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Requested vote from peers 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:50.991302 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "badb79f796d240029f0d8bc90deaa5c6" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:50.991294 2483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "badb79f796d240029f0d8bc90deaa5c6" candidate_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0103de40bc4724a20d7dc4bff02114"
I20250905 07:50:50.991475 2339 raft_consensus.cc:3058] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:50.991479 2483 raft_consensus.cc:3058] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:50.993444 2339 raft_consensus.cc:2466] T badb79f796d240029f0d8bc90deaa5c6 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:50.993818 2552 leader_election.cc:304] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715f96a2595d4c4e9eb0d68aea3465f4, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:50.994066 4010 raft_consensus.cc:2802] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:50.994441 2483 raft_consensus.cc:2466] T badb79f796d240029f0d8bc90deaa5c6 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715f96a2595d4c4e9eb0d68aea3465f4 in term 1.
I20250905 07:50:50.994871 4010 raft_consensus.cc:695] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 LEADER]: Becoming Leader. State: Replica: 715f96a2595d4c4e9eb0d68aea3465f4, State: Running, Role: LEADER
I20250905 07:50:50.995245 4010 consensus_queue.cc:237] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
W20250905 07:50:51.014485 2669 tablet.cc:2378] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:50:51.037914 2388 tablet.cc:2378] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:50:51.037968 2531 tablet.cc:2378] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 07:50:51.110133 2273 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:51.110203 2553 connection.cc:537] server connection from 127.0.0.1:60258 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:51.110631 2416 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:51.110968 2673 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:51.111157 2552 connection.cc:537] client connection to 127.1.178.190:35357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 07:50:51.111974 2673 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.178.190:35357)
W20250905 07:50:51.112591 2673 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:51.112906 2673 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.178.190:35357)
W20250905 07:50:51.113642 2673 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.178.190:35357)
I20250905 07:50:51.155881 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:50:51.156109 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:51.299252 4306 raft_consensus.cc:491] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:51.299458 4306 raft_consensus.cc:513] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:51.300316 4306 leader_election.cc:290] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:51.300904 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b1fb1dd3cb94ca09890a94f8f1d0f43" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" is_pre_election: true
I20250905 07:50:51.300918 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b1fb1dd3cb94ca09890a94f8f1d0f43" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e" is_pre_election: true
I20250905 07:50:51.301136 2619 raft_consensus.cc:2466] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:51.301159 2339 raft_consensus.cc:2466] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 0.
I20250905 07:50:51.301595 2416 leader_election.cc:304] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:51.301878 4306 raft_consensus.cc:2802] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 07:50:51.301978 4306 raft_consensus.cc:491] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 07:50:51.302063 4306 raft_consensus.cc:3058] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:51.303963 4306 raft_consensus.cc:513] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:51.304477 4306 leader_election.cc:290] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Requested vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773)
I20250905 07:50:51.305008 2619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b1fb1dd3cb94ca09890a94f8f1d0f43" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715f96a2595d4c4e9eb0d68aea3465f4"
I20250905 07:50:51.305037 2339 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b1fb1dd3cb94ca09890a94f8f1d0f43" candidate_uuid: "0f0103de40bc4724a20d7dc4bff02114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b902ef48db944a02813b8543b2f91a7e"
I20250905 07:50:51.305182 2619 raft_consensus.cc:3058] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:51.305220 2339 raft_consensus.cc:3058] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Advancing to term 1
I20250905 07:50:51.307101 2339 raft_consensus.cc:2466] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P b902ef48db944a02813b8543b2f91a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:51.307204 2619 raft_consensus.cc:2466] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 715f96a2595d4c4e9eb0d68aea3465f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0103de40bc4724a20d7dc4bff02114 in term 1.
I20250905 07:50:51.307545 2416 leader_election.cc:304] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0103de40bc4724a20d7dc4bff02114, b902ef48db944a02813b8543b2f91a7e; no voters:
I20250905 07:50:51.307843 4306 raft_consensus.cc:2802] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 07:50:51.308012 4306 raft_consensus.cc:695] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0103de40bc4724a20d7dc4bff02114, State: Running, Role: LEADER
I20250905 07:50:51.308364 4306 consensus_queue.cc:237] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } }
I20250905 07:50:51.344208 4288 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:51.344415 4288 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:51.345141 4288 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:50:51.346045 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:50:51.346248 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:50:51.346395 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:50:51.346671 4288 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:50:51.378026 4308 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:51.378334 4308 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:51.379144 4308 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:51.380970 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:51.381147 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:51.381289 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:51.381541 4308 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:50:51.857486 4010 consensus_queue.cc:1035] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:51.868613 4313 consensus_queue.cc:1035] T badb79f796d240029f0d8bc90deaa5c6 P 715f96a2595d4c4e9eb0d68aea3465f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:51.996472 4322 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:51.996678 4322 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:51.997485 4322 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:51.998281 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250905 07:50:51.999346 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:51.999517 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:51.999645 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:51.999893 4322 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:50:52.053037 1738 master-stress-test.cc:577] Killing test thread 1/25
I20250905 07:50:52.177783 4305 consensus_queue.cc:1035] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:52.183326 4305 consensus_queue.cc:1035] T 6b1fb1dd3cb94ca09890a94f8f1d0f43 P 0f0103de40bc4724a20d7dc4bff02114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 07:50:52.850961 4337 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:52.851186 4337 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:52.851974 4337 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:52.854230 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:52.854453 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:52.854584 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:52.854861 4337 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:50:53.496646 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 6: this message will repeat every 5th retry.
W20250905 07:50:53.580947 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 6: this message will repeat every 5th retry.
I20250905 07:50:54.290208 4359 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:54.290408 4359 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:54.290987 4359 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:54.293031 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:54.293198 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:54.293336 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:54.293594 4359 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:50:56.157559 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:50:56.157776 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:50:56.438669 4396 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:56.438891 4396 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:56.439612 4396 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:56.441442 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:56.441605 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:56.441737 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:56.441999 4396 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:50:56.593890 4400 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:50:56.594125 4400 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:50:56.594772 4400 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:50:56.595739 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:50:56.595983 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:50:56.596072 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:50:56.596344 4400 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:50:57.292544 4411 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:57.292776 4411 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:57.293449 4411 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:57.294361 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250905 07:50:57.295487 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:50:57.295656 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:50:57.295797 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:57.296082 4411 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:50:58.159514 4426 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:50:58.159710 4426 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:50:58.160467 4426 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:50:58.162431 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:50:58.162626 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:50:58.162750 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:50:58.162997 4426 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:51:01.159314 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:01.159503 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:51:01.716118 4480 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:51:01.716356 4480 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:51:01.717110 4480 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:51:01.719202 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:51:01.719393 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:51:01.719518 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:51:01.719791 4480 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:51:02.978312 4497 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:51:02.978525 4497 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:51:02.979405 4497 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:51:02.980367 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:51:02.980583 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:51:02.980690 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:51:02.980957 4497 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:51:03.524058 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 11: this message will repeat every 5th retry.
W20250905 07:51:03.809708 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 11: this message will repeat every 5th retry.
I20250905 07:51:06.161146 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:06.161365 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:51:11.162901 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:11.163122 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:51:12.418244 4628 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:51:12.418474 4628 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:51:12.419111 4628 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:51:12.420126 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:51:12.420361 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:51:12.420464 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:51:12.420739 4628 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:51:13.238557 4646 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:51:13.238760 4646 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:51:13.239439 4646 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:51:13.240558 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250905 07:51:13.241838 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:51:13.242033 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:51:13.242156 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:51:13.242436 4646 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:51:13.557628 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 16: this message will repeat every 5th retry.
W20250905 07:51:14.164317 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 16: this message will repeat every 5th retry.
I20250905 07:51:16.164701 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:16.164913 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:51:18.489369 4729 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:51:18.489590 4729 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:51:18.490386 4729 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:51:18.491495 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:51:18.492831 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:51:18.493024 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:51:18.493135 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:51:18.493412 4729 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:51:18.776137 2386 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (59 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:51:19.055717 2528 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (141 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:51:19.086241 2667 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (60 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:51:21.166464 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:21.166684 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:51:23.741303 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 21: this message will repeat every 5th retry.
W20250905 07:51:24.634766 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 21: this message will repeat every 5th retry.
I20250905 07:51:25.081008 4827 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:51:25.081220 4827 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:51:25.082116 4827 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:51:25.083048 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:51:25.083271 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:51:25.083390 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:51:25.083670 4827 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:51:26.168051 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:26.168242 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:51:27.732949 4874 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:51:27.733157 4874 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:51:27.733865 4874 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:51:27.736920 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:51:27.738147 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:51:27.738310 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:51:27.738471 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:51:27.738765 4874 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:51:31.169675 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:31.169898 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:51:33.597283 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 26: this message will repeat every 5th retry.
W20250905 07:51:35.850903 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 26: this message will repeat every 5th retry.
I20250905 07:51:36.171604 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:36.171805 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:51:37.974907 5025 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:51:37.975128 5025 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:51:37.975832 5025 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:51:37.976842 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:51:37.977092 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:51:37.977186 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:51:37.977487 5025 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:51:39.295496 5043 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:51:39.295708 5043 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:51:39.296448 5043 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:51:39.297542 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:51:39.297550 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:51:39.298849 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:51:39.299024 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:51:39.299146 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:51:39.299425 5043 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:51:41.173475 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:41.173712 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:51:43.386915 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 31: this message will repeat every 5th retry.
I20250905 07:51:46.175420 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:46.175648 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:51:46.257772 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 31: this message will repeat every 5th retry.
I20250905 07:51:50.846276 5211 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:51:50.846609 5211 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:51:50.847411 5211 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:51:50.848694 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:51:50.848721 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:51:50.849993 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:51:50.850134 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:51:50.850244 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:51:50.850481 5211 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:51:51.176865 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:51.177070 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:51:51.426879 5225 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:51:51.427099 5225 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:51:51.427830 5225 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:51:51.429558 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:51:51.429754 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:51:51.429876 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:51:51.430114 5225 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:51:53.248845 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 36: this message will repeat every 5th retry.
I20250905 07:51:53.284672 5244 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:51:53.284883 5244 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:51:53.285835 5244 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:51:53.286993 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:51:53.287189 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:51:53.287283 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:51:53.287542 5244 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:51:56.089540 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 36: this message will repeat every 5th retry.
I20250905 07:51:56.178571 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:51:56.178773 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:01.180457 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:01.180666 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:02.916746 5380 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:52:02.916981 5380 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:52:02.917811 5380 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:52:02.919024 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:52:02.919257 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:52:02.919345 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:52:02.919600 5380 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:52:04.135732 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 41: this message will repeat every 5th retry.
W20250905 07:52:06.123483 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 41: this message will repeat every 5th retry.
I20250905 07:52:06.182160 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:06.182376 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:06.304275 5439 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:52:06.304513 5439 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:52:06.305233 5439 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:52:06.305840 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20250905 07:52:06.307089 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:52:06.307255 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:52:06.307427 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:52:06.307696 5439 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:52:11.183938 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:11.184134 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:52:14.082664 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 46: this message will repeat every 5th retry.
W20250905 07:52:15.407451 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 46: this message will repeat every 5th retry.
I20250905 07:52:16.107017 5596 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:52:16.107220 5596 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:52:16.107873 5596 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:52:16.109066 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:52:16.110312 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:52:16.110492 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:52:16.110608 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:52:16.110877 5596 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:52:16.190161 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:16.190405 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:17.704262 5620 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:52:17.704501 5620 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:52:17.705157 5620 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:52:17.707392 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:52:17.707579 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:52:17.707710 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:52:17.707988 5620 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:52:18.969877 2386 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (179 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:52:19.274156 2667 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (180 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:52:19.274173 2666 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (223 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:52:19.555712 2528 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (325 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:52:21.073928 5666 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:52:21.074158 5666 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:52:21.075134 5666 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:52:21.076118 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:52:21.076354 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:52:21.076478 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:52:21.076761 5666 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:52:21.192231 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:21.192507 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:52:23.596740 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 51: this message will repeat every 5th retry.
W20250905 07:52:24.924954 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 51: this message will repeat every 5th retry.
I20250905 07:52:26.194162 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:26.194416 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:31.195993 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:31.196202 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:31.599043 5820 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:52:31.599288 5820 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:52:31.600175 5820 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:52:31.601081 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:52:31.601414 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:52:31.601523 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:52:31.601812 5820 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:52:33.488356 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 56: this message will repeat every 5th retry.
W20250905 07:52:35.238566 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 56: this message will repeat every 5th retry.
I20250905 07:52:35.262643 5867 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:52:35.262845 5867 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:52:35.263502 5867 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:52:35.264441 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:52:35.264614 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:52:35.264703 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:52:35.264941 5867 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:52:35.554566 5880 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:52:35.554759 5880 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:52:35.555696 5880 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:52:35.556774 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250905 07:52:35.558020 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:52:35.558171 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:52:35.558333 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:52:35.558607 5880 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:52:36.197674 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:36.197871 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:40.112287 5948 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:52:40.112519 5948 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:52:40.113185 5948 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:52:40.115406 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:52:40.115708 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:52:40.115859 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:52:40.116181 5948 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:52:41.199637 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:41.199841 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:52:44.700635 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 61: this message will repeat every 5th retry.
W20250905 07:52:44.738880 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 61: this message will repeat every 5th retry.
I20250905 07:52:46.201206 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:46.201395 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:46.210678 4033 maintenance_manager.cc:419] P 65033f3e4b44455389e29a5e1c9f82cd: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033354
I20250905 07:52:46.268329 4032 maintenance_manager.cc:643] P 65033f3e4b44455389e29a5e1c9f82cd: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.057s user 0.026s sys 0.004s Metrics: {"bytes_written":16898,"cfile_init":1,"dirs.queue_time_us":433,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":967,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":37,"thread_start_us":592,"threads_started":2,"wal-append.queue_time_us":462}
I20250905 07:52:46.269654 4033 maintenance_manager.cc:419] P 65033f3e4b44455389e29a5e1c9f82cd: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1947 bytes on disk
I20250905 07:52:46.270434 4032 maintenance_manager.cc:643] P 65033f3e4b44455389e29a5e1c9f82cd: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20250905 07:52:51.202952 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:51.203176 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:51.673298 6126 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:52:51.673519 6126 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:52:51.674439 6126 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:52:51.675290 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:52:51.675539 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:52:51.675639 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:52:51.675959 6126 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:52:54.084257 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 66: this message will repeat every 5th retry.
W20250905 07:52:54.476941 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 66: this message will repeat every 5th retry.
I20250905 07:52:55.474541 6170 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:52:55.474769 6170 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:52:55.475543 6170 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:52:55.476537 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:52:55.476801 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:52:55.476902 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:52:55.477200 6170 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:52:56.204811 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:52:56.205010 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:52:57.385862 6210 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:52:57.386116 6210 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:52:57.387063 6210 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:52:57.388091 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250905 07:52:57.389312 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:52:57.389488 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:52:57.389631 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:52:57.389892 6210 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:53:01.206604 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:01.206857 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:53:04.503638 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 71: this message will repeat every 5th retry.
W20250905 07:53:04.623097 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 71: this message will repeat every 5th retry.
I20250905 07:53:06.208407 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:06.208607 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:11.210139 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:11.210343 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:11.403128 6471 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:53:11.403393 6471 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:53:11.404254 6471 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:53:11.404821 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:53:11.406750 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:53:11.406944 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:53:11.407044 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:53:11.407281 6471 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:53:11.450662 6462 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:11.450851 6462 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:53:11.451644 6462 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:53:11.452611 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:53:11.452883 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:53:11.452973 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:53:11.453270 6462 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:53:13.931731 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 76: this message will repeat every 5th retry.
W20250905 07:53:14.136327 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 76: this message will repeat every 5th retry.
I20250905 07:53:16.212023 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:16.212203 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:53:19.174388 2386 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (299 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:53:19.474053 2666 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (343 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:53:19.851577 2528 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (508 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:53:21.213799 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:21.213997 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:53:23.999567 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 81: this message will repeat every 5th retry.
W20250905 07:53:24.152556 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 81: this message will repeat every 5th retry.
I20250905 07:53:26.215787 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:26.216022 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:27.035837 6726 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:27.036098 6726 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:53:27.036957 6726 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:53:27.037998 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:53:27.038230 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:53:27.038337 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:53:27.038617 6726 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:53:30.268617 6779 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:53:30.268810 6779 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:53:30.269639 6779 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:53:30.270545 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:53:30.271788 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:53:30.271979 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:53:30.272116 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:53:30.272356 6779 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:53:31.217499 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:31.217720 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:32.614972 6816 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:32.615185 6816 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:53:32.615882 6816 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:53:32.616874 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:53:32.617069 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:53:32.617146 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:53:32.617398 6816 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:53:33.763360 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 86: this message will repeat every 5th retry.
W20250905 07:53:34.698863 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 86: this message will repeat every 5th retry.
I20250905 07:53:36.219358 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:36.219609 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:36.381053 6891 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:53:36.381276 6891 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:53:36.381937 6891 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:53:36.382566 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:53:36.383845 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:53:36.384042 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:53:36.384168 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:53:36.384398 6891 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:53:41.221262 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:41.221495 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:42.169700 6992 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:42.169934 6992 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:53:42.170699 6992 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:53:42.171682 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:53:42.171864 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:53:42.171970 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:53:42.172206 6992 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:53:43.232208 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 91: this message will repeat every 5th retry.
W20250905 07:53:44.577293 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 91: this message will repeat every 5th retry.
I20250905 07:53:44.929605 7048 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:53:44.929838 7048 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:53:44.930552 7048 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:53:44.931558 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:53:44.932698 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:53:44.932919 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:53:44.933034 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:53:44.933305 7048 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:53:46.223887 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:46.224119 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:48.757196 7118 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:48.757418 7118 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:53:48.758090 7118 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:53:48.759032 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:53:48.759239 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:53:48.759326 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:53:48.759575 7118 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:53:49.190254 7133 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:53:49.190454 7133 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:53:49.191113 7133 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:53:49.193193 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:53:49.193411 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:53:49.193526 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:53:49.193794 7133 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:53:51.225338 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:51.225541 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:53:53.575553 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 96: this message will repeat every 5th retry.
W20250905 07:53:54.203642 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 96: this message will repeat every 5th retry.
I20250905 07:53:55.344022 7237 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:53:55.344280 7237 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:53:55.345019 7237 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:53:55.346122 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:53:55.346364 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:53:55.346475 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:53:55.346796 7237 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:53:56.227006 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:53:56.227193 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:53:56.757997 7266 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:53:56.758206 7266 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:53:56.758898 7266 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:53:56.759897 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250905 07:53:56.761258 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:53:56.761431 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:53:56.761557 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:53:56.761842 7266 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:54:01.228916 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:01.229182 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:54:03.164948 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 101: this message will repeat every 5th retry.
W20250905 07:54:03.859412 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 101: this message will repeat every 5th retry.
I20250905 07:54:05.541765 7413 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:05.541983 7413 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:54:05.542945 7413 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:54:05.543955 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:54:05.544220 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:54:05.544313 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:54:05.544610 7413 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:54:06.230790 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:06.230978 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:54:11.232779 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:11.232990 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:54:12.619628 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 106: this message will repeat every 5th retry.
W20250905 07:54:12.689491 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 106: this message will repeat every 5th retry.
I20250905 07:54:14.167155 7575 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:54:14.167378 7575 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:54:14.168351 7575 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:54:14.169412 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:54:14.170732 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:54:14.170912 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:54:14.171039 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:54:14.171293 7575 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:54:15.229219 7592 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:15.229455 7592 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:54:15.230161 7592 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:54:15.231235 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:54:15.231437 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:54:15.231527 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:54:15.231765 7592 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:54:16.234493 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:16.234696 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:54:18.053385 7647 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:54:18.053622 7647 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:54:18.054262 7647 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:54:18.056296 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:54:18.056532 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:54:18.056731 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:54:18.056985 7647 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:54:19.153407 7666 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:19.153640 7666 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:54:19.154469 7666 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:54:19.155083 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:54:19.155469 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:54:19.155572 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:54:19.155798 7666 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:54:19.358963 2386 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (419 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:54:19.671952 2666 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (463 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:54:20.015450 2528 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (691 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:54:21.236090 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:21.236301 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:54:22.056725 7719 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:54:22.056967 7719 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:54:22.057662 7719 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:54:22.058670 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250905 07:54:22.060107 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:54:22.060264 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:54:22.060397 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:54:22.060663 7719 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:54:22.408427 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 111: this message will repeat every 5th retry.
W20250905 07:54:22.442276 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 111: this message will repeat every 5th retry.
I20250905 07:54:26.237812 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:26.238018 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:54:31.239585 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:31.239822 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:54:32.461999 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 116: this message will repeat every 5th retry.
W20250905 07:54:32.839017 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 116: this message will repeat every 5th retry.
I20250905 07:54:33.327847 7890 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:54:33.328135 7890 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:54:33.328925 7890 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:54:33.330114 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:54:33.331882 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:54:33.332100 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:54:33.332253 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:54:33.332536 7890 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:54:33.890003 7903 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:33.890218 7903 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:54:33.891216 7903 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:54:33.892202 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:54:33.892414 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:54:33.892509 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:54:33.892732 7903 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:54:36.241276 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:36.241487 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:54:39.913147 7980 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:54:39.913403 7980 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:54:39.914150 7980 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:54:39.915135 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:54:39.916576 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:54:39.916723 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:54:39.916860 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:54:39.917117 7980 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:54:41.243029 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:41.243276 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:54:42.276084 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 121: this message will repeat every 5th retry.
W20250905 07:54:43.313833 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 121: this message will repeat every 5th retry.
I20250905 07:54:45.444068 8068 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:54:45.444288 8068 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:54:45.444941 8068 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:54:45.445959 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:54:45.447033 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:54:45.447217 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:54:45.447338 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:54:45.447574 8068 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:54:46.244526 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:46.244724 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:54:46.513296 8085 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:46.513500 8085 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:54:46.514149 8085 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:54:46.515157 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:54:46.515372 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:54:46.515466 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:54:46.515712 8085 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:54:49.404486 8138 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:54:49.404713 8138 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:54:49.405494 8138 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:54:49.407472 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:54:49.407656 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:54:49.407778 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:54:49.408107 8138 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:54:51.246327 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:51.246526 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:54:51.489042 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 126: this message will repeat every 5th retry.
W20250905 07:54:53.540437 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 126: this message will repeat every 5th retry.
I20250905 07:54:56.248273 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:54:56.248492 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:54:59.423128 8309 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:54:59.423321 8309 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:54:59.424185 8309 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:54:59.425185 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:54:59.425377 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:54:59.425464 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:54:59.425688 8309 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:55:01.250157 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:01.250412 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:01.285259 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 131: this message will repeat every 5th retry.
I20250905 07:55:03.294620 8380 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:55:03.294880 8380 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:55:03.295535 8380 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:55:03.296591 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:03.296591 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250905 07:55:03.297833 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:03.298009 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:55:03.298098 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:55:03.298319 8380 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:55:03.632678 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 131: this message will repeat every 5th retry.
I20250905 07:55:06.252043 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:06.252241 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:55:08.886615 8474 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:55:08.886837 8474 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:55:08.887437 8474 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:55:08.888389 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:55:08.889794 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:55:08.890015 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:55:08.890131 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:55:08.890398 8474 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:55:10.824307 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 136: this message will repeat every 5th retry.
I20250905 07:55:11.253736 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:11.253957 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:13.141036 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 136: this message will repeat every 5th retry.
I20250905 07:55:15.544487 8594 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:55:15.544759 8594 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:55:15.546070 8594 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:55:15.546932 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:55:15.547139 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:55:15.547225 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:55:15.547503 8594 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:55:16.255398 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:16.255601 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:19.567241 2386 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (539 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:19.845142 2667 heartbeater.cc:646] Failed to heartbeat to 127.1.178.190:35357 (540 consecutive failures): Network error: Failed to ping master at 127.1.178.190:35357: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:20.442305 2528 heartbeater.cc:646] Failed to heartbeat to 127.1.178.188:39743 (873 consecutive failures): Network error: Failed to ping master at 127.1.178.188:39743: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:55:20.442804 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 141: this message will repeat every 5th retry.
I20250905 07:55:21.257184 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:21.257404 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:23.242225 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 141: this message will repeat every 5th retry.
I20250905 07:55:23.297991 8718 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:55:23.298214 8718 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:55:23.298844 8718 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:55:23.299804 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:55:23.300020 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:55:23.300118 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:55:23.300341 8718 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:55:23.868125 8734 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:55:23.868350 8734 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:55:23.869045 8734 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:55:23.870141 4030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:55:23.871441 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:23.871651 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:55:23.871802 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:55:23.872118 8734 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:55:25.214231 8763 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:55:25.214452 8763 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:55:25.215086 8763 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:55:25.217242 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:25.217429 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:55:25.217578 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:55:25.217837 8763 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:55:26.258833 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:26.259059 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:30.265843 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 146: this message will repeat every 5th retry.
I20250905 07:55:30.884637 8864 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:55:30.884838 8864 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:55:30.885412 8864 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:55:30.886281 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250905 07:55:30.887465 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:30.887630 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:55:30.887782 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:55:30.888051 8864 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:55:31.260219 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:31.260419 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:32.860970 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 146: this message will repeat every 5th retry.
I20250905 07:55:36.261991 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:36.262202 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:40.233337 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 151: this message will repeat every 5th retry.
I20250905 07:55:40.585444 9026 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:55:40.585652 9026 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:55:40.586393 9026 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:55:40.587450 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:55:40.587641 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:55:40.587715 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:55:40.587996 9026 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 07:55:41.263784 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:41.264030 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:55:43.521190 9081 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:55:43.521381 9081 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:55:43.522028 9081 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:55:43.522967 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:55:43.524381 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
W20250905 07:55:43.524585 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
I20250905 07:55:43.524706 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:55:43.524981 9081 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250905 07:55:43.812338 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer b902ef48db944a02813b8543b2f91a7e (127.1.178.129:45773): Couldn't send request to peer b902ef48db944a02813b8543b2f91a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 151: this message will repeat every 5th retry.
I20250905 07:55:46.265707 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:46.265949 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 07:55:47.713999 9145 raft_consensus.cc:491] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 07:55:47.714190 9145 raft_consensus.cc:513] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715f96a2595d4c4e9eb0d68aea3465f4" member_type: VOTER last_known_addr { host: "127.1.178.131" port: 45275 } } peers { permanent_uuid: "0f0103de40bc4724a20d7dc4bff02114" member_type: VOTER last_known_addr { host: "127.1.178.130" port: 37221 } } peers { permanent_uuid: "b902ef48db944a02813b8543b2f91a7e" member_type: VOTER last_known_addr { host: "127.1.178.129" port: 45773 } }
I20250905 07:55:47.714787 9145 leader_election.cc:290] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275), 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221)
W20250905 07:55:47.715554 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
W20250905 07:55:47.715727 2273 leader_election.cc:343] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f0103de40bc4724a20d7dc4bff02114 (127.1.178.130:37221): Not found: Tablet not found: b1e22c8f127247f0b26025c3d95937d3
I20250905 07:55:47.715785 2273 leader_election.cc:304] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b902ef48db944a02813b8543b2f91a7e; no voters: 0f0103de40bc4724a20d7dc4bff02114, 715f96a2595d4c4e9eb0d68aea3465f4
I20250905 07:55:47.715999 9145 raft_consensus.cc:2747] T b1e22c8f127247f0b26025c3d95937d3 P b902ef48db944a02813b8543b2f91a7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 07:55:50.029203 2416 consensus_peers.cc:489] T f08841e4d54446699cd387180cdd871a P 0f0103de40bc4724a20d7dc4bff02114 -> Peer 715f96a2595d4c4e9eb0d68aea3465f4 (127.1.178.131:45275): Couldn't send request to peer 715f96a2595d4c4e9eb0d68aea3465f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08841e4d54446699cd387180cdd871a. This is attempt 156: this message will repeat every 5th retry.
I20250905 07:55:50.996057 9218 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9dbf15725a7342da830f57c0535eb72d)
I20250905 07:55:50.996276 9218 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf15725a7342da830f57c0535eb72d" member_type: VOTER last_known_addr { host: "127.1.178.190" port: 35357 } } peers { permanent_uuid: "65033f3e4b44455389e29a5e1c9f82cd" member_type: VOTER last_known_addr { host: "127.1.178.189" port: 43391 } } peers { permanent_uuid: "81e2a8aedb144e61891de4dcd390ff35" member_type: VOTER last_known_addr { host: "127.1.178.188" port: 39743 } }
I20250905 07:55:50.996974 9218 leader_election.cc:290] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357), 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743)
W20250905 07:55:50.998034 4028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 07:55:50.999228 4030 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 81e2a8aedb144e61891de4dcd390ff35 (127.1.178.188:39743): Network error: Client connection negotiation failed: client connection to 127.1.178.188:39743: connect: Connection refused (error 111)
W20250905 07:55:50.999395 4028 leader_election.cc:336] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9dbf15725a7342da830f57c0535eb72d (127.1.178.190:35357): Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111)
I20250905 07:55:50.999527 4028 leader_election.cc:304] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65033f3e4b44455389e29a5e1c9f82cd; no voters: 81e2a8aedb144e61891de4dcd390ff35, 9dbf15725a7342da830f57c0535eb72d
I20250905 07:55:50.999727 9218 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 65033f3e4b44455389e29a5e1c9f82cd [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250905 07:55:51.083369 4033 maintenance_manager.cc:419] P 65033f3e4b44455389e29a5e1c9f82cd: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1947 bytes on disk
I20250905 07:55:51.089988 4032 maintenance_manager.cc:643] P 65033f3e4b44455389e29a5e1c9f82cd: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.002s sys 0.004s Metrics: {"dirs.queue_time_us":491,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":91,"thread_start_us":771,"threads_started":2,"wal-append.queue_time_us":531}
I20250905 07:55:51.267170 4012 master_runner.cc:186] Error getting master registration for 127.1.178.190:35357: Network error: Client connection negotiation failed: client connection to 127.1.178.190:35357: connect: Connection refused (error 111),
I20250905 07:55:51.267369 4012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 07:55:51.671079 2673 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
W20250905 07:55:51.671326 2673 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
W20250905 07:55:51.671448 2673 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
W20250905 07:55:51.671589 2673 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
F20250905 07:55:51.682401 2707 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
*** Aborted at 1757058951 (unix time) try "date -d @1757058951" if you are using GNU date ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000006ca) received by PID 1738 (TID 0x7fa935115700) from PID 1738; stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690637 2691 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690637 2691 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690786 2690 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7c0359e055804d0b9a84bd4dd5e60634 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7fa954404980 (unknown) at ??:0
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690637 2691 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690786 2690 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7c0359e055804d0b9a84bd4dd5e60634 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.691920 2701 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690637 2691 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690786 2690 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7c0359e055804d0b9a84bd4dd5e60634 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.691920 2701 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692003 2697 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7fa94d397fb7 gsignal at ??:0
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690637 2691 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690786 2690 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7c0359e055804d0b9a84bd4dd5e60634 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.691920 2701 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692003 2697 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692414 2686 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4907c91834874a1a88a3312b6a0be96f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690637 2691 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690786 2690 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7c0359e055804d0b9a84bd4dd5e60634 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.691920 2701 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692003 2697 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692414 2686 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4907c91834874a1a88a3312b6a0be96f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692488 2685 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b3cf8c80578f4b4f84327aedafee6a93 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
F20250905 07:55:51.683774 2695 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.683775 2694 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.684808 2706 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685045 2688 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6a86ab58c8f4eddbf3e0c4edf058247 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685228 2681 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f62349a373e45b8ae063dafbfc3cdcc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685282 2683 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cd8e6f84c6614cd58e55c59c90383205 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.685565 2698 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686460 2692 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.686586 2702 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687857 2682 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6abbacd07c2f4ab4b74761cd100dcf58 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.687867 2693 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688213 2689 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3daa55fc9e5447f82536064de82c58c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.688230 2687 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_347a8cac73e54038b34319425f8f3bd2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689288 2699 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.689426 2684 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06530af7099e45c7b150408f8c1a3436 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690629 2696 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690637 2691 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.690786 2690 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7c0359e055804d0b9a84bd4dd5e60634 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.691920 2701 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692003 2697 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692414 2686 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4907c91834874a1a88a3312b6a0be96f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692488 2685 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b3cf8c80578f4b4f84327aedafee6a93 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)F20250905 07:55:51.692829 2703 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.178.190:35357,127.1.178.189:43391,127.1.178.188:39743, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7fa94d399921 abort at ??:0
@ 0x7fa9500cfdcd google::LogMessage::Fail() at ??:0
@ 0x7fa9500d3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa9500cf7cc google::LogMessage::Flush() at ??:0
@ 0x7fa9500d0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40d9f5 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:391
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fa94da1f6df (unknown) at ??:0
@ 0x7fa9543f96db start_thread at ??:0
@ 0x7fa94d47a71f clone at ??:0