Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250903 14:22:41.381004 416 test_util.cc:276] Using random seed: 1449344603
2025-09-03T14:22:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T14:22:41Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklEwXqn/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklEwXqn/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/security.properties
2025-09-03 14:22:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklEwXqn/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklEwXqn/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250903 14:22:58.140854 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 14:22:58.537370 660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:22:58.537813 660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:22:58.537954 660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:22:58.547000 660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:22:58.547135 660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:22:58.547226 660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:22:58.547299 660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:22:58.559099 660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:22:58.560937 660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:22:58.562939 660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:22:58.573347 666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:22:58.574048 668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:22:58.574563 665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:22:58.575336 660 server_base.cc:1047] running on GCE node
I20250903 14:22:58.576294 660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:22:58.577984 660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:22:58.579206 660 hybrid_clock.cc:648] HybridClock initialized: now 1756909378579156 us; error 51 us; skew 500 ppm
I20250903 14:22:58.579641 660 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:22:58.582156 660 webserver.cc:480] Webserver started at http://127.0.104.62:33279/ using document root <none> and password file <none>
I20250903 14:22:58.582968 660 fs_manager.cc:362] Metadata directory not provided
I20250903 14:22:58.583091 660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:22:58.583506 660 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:22:58.586076 660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/instance:
uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
format_stamp: "Formatted at 2025-09-03 14:22:58 on dist-test-slave-rxtz"
I20250903 14:22:58.586915 660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal/instance:
uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
format_stamp: "Formatted at 2025-09-03 14:22:58 on dist-test-slave-rxtz"
I20250903 14:22:58.592955 660 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250903 14:22:58.597606 674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:22:58.599530 660 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20250903 14:22:58.599735 660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
format_stamp: "Formatted at 2025-09-03 14:22:58 on dist-test-slave-rxtz"
I20250903 14:22:58.599951 660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:22:58.628298 660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:22:58.629307 660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:22:58.629637 660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:22:58.653168 660 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37899
I20250903 14:22:58.653206 725 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37899 every 8 connection(s)
I20250903 14:22:58.654829 660 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
I20250903 14:22:58.659051 726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:22:58.663700 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 660
I20250903 14:22:58.664008 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal/instance
I20250903 14:22:58.665367 726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:22:58.666630 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 14:22:58.684206 726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
W20250903 14:22:58.685309 676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:22:58.686884 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
I20250903 14:22:58.736248 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } attempt: 1
W20250903 14:22:58.738370 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
I20250903 14:22:58.806627 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } attempt: 2
W20250903 14:22:58.808682 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
I20250903 14:22:58.899986 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } attempt: 3
W20250903 14:22:58.901944 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:22:59.042037 729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:22:59.042414 729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:22:59.042537 729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250903 14:22:59.045202 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } attempt: 4
W20250903 14:22:59.047241 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:22:59.052892 729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:22:59.053032 729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:22:59.053105 729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:22:59.053154 729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:22:59.063793 729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:22:59.065670 729 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:22:59.067934 729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:22:59.078995 735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:22:59.078995 738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:22:59.079051 736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:22:59.080137 729 server_base.cc:1047] running on GCE node
I20250903 14:22:59.081100 729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:22:59.082846 729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:22:59.084259 729 hybrid_clock.cc:648] HybridClock initialized: now 1756909379084177 us; error 96 us; skew 500 ppm
I20250903 14:22:59.084704 729 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:22:59.088732 729 webserver.cc:480] Webserver started at http://127.0.104.61:43811/ using document root <none> and password file <none>
I20250903 14:22:59.089797 729 fs_manager.cc:362] Metadata directory not provided
I20250903 14:22:59.089948 729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:22:59.090533 729 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:22:59.094923 729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/instance:
uuid: "f4c58d929e53460883d05e0f14f20780"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:22:59.096138 729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal/instance:
uuid: "f4c58d929e53460883d05e0f14f20780"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:22:59.104445 729 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20250903 14:22:59.110013 744 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:22:59.113160 729 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250903 14:22:59.113417 729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
uuid: "f4c58d929e53460883d05e0f14f20780"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:22:59.113708 729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:22:59.149859 729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:22:59.150874 729 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:22:59.151161 729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:22:59.175531 729 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:40437
I20250903 14:22:59.175549 795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:40437 every 8 connection(s)
I20250903 14:22:59.177162 729 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
I20250903 14:22:59.177598 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 729
I20250903 14:22:59.177762 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal/instance
I20250903 14:22:59.178891 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43143
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 14:22:59.181835 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:22:59.189131 796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:22:59.205085 796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:22:59.217453 796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } has no permanent_uuid. Determining permanent_uuid...
W20250903 14:22:59.218799 746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:22:59.220715 796 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:22:59.270179 796 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 1
W20250903 14:22:59.272132 796 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:22:59.340454 796 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 2
W20250903 14:22:59.342365 796 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:22:59.346534 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } attempt: 5
I20250903 14:22:59.352918 726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } has no permanent_uuid. Determining permanent_uuid...
W20250903 14:22:59.354730 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:22:59.406045 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 1
W20250903 14:22:59.408155 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:22:59.433651 796 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 3
W20250903 14:22:59.435566 796 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:22:59.476405 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 2
W20250903 14:22:59.478420 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:22:59.555586 797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:22:59.555935 797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:22:59.556072 797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:22:59.564977 797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:22:59.565096 797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:22:59.565168 797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:22:59.565232 797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:22:59.575852 797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:22:59.577596 797 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:22:59.578876 796 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 4
I20250903 14:22:59.579836 797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:22:59.580741 796 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:22:59.584704 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 3
W20250903 14:22:59.586959 726 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:22:59.589545 807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:22:59.590340 806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:22:59.590337 809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:22:59.590662 797 server_base.cc:1047] running on GCE node
I20250903 14:22:59.591704 797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:22:59.593261 797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:22:59.594491 797 hybrid_clock.cc:648] HybridClock initialized: now 1756909379594440 us; error 52 us; skew 500 ppm
I20250903 14:22:59.594852 797 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:22:59.597332 797 webserver.cc:480] Webserver started at http://127.0.104.60:41549/ using document root <none> and password file <none>
I20250903 14:22:59.598147 797 fs_manager.cc:362] Metadata directory not provided
I20250903 14:22:59.598258 797 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:22:59.598690 797 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:22:59.601368 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/instance:
uuid: "2bbe08ab62bf4073825bf1b598ad2edb"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:22:59.602226 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal/instance:
uuid: "2bbe08ab62bf4073825bf1b598ad2edb"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:22:59.608284 797 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250903 14:22:59.612284 815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:22:59.613893 797 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:22:59.614105 797 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
uuid: "2bbe08ab62bf4073825bf1b598ad2edb"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:22:59.614322 797 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:22:59.632581 797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:22:59.633790 797 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:22:59.634132 797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:22:59.656646 797 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:43143
I20250903 14:22:59.656702 866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:43143 every 8 connection(s)
I20250903 14:22:59.658313 797 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/info.pb
I20250903 14:22:59.663213 867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:22:59.668926 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 797
I20250903 14:22:59.669166 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal/instance
I20250903 14:22:59.672945 867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:22:59.699688 867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:22:59.708355 867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } has no permanent_uuid. Determining permanent_uuid...
I20250903 14:22:59.720332 867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Bootstrap starting.
I20250903 14:22:59.724839 867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Neither blocks nor log segments found. Creating new log.
I20250903 14:22:59.726037 867 log.cc:826] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Log is configured to *not* fsync() on all Append() calls
I20250903 14:22:59.728783 867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: No bootstrap required, opened a new log
I20250903 14:22:59.734019 867 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:22:59.734345 867 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:22:59.734422 867 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe08ab62bf4073825bf1b598ad2edb, State: Initialized, Role: FOLLOWER
I20250903 14:22:59.735030 867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:22:59.735977 875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:22:59.736424 875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:22:59.736972 867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: configured and running, proceeding with master startup.
W20250903 14:22:59.752034 889 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:22:59.752194 889 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:22:59.751984 797 master_runner.cc:186] Error getting master registration for 127.0.104.62:37899: OK, instance_id { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" instance_seqno: 1756909378646681 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250903 14:22:59.756726 797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:22:59.764230 726 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 4
I20250903 14:22:59.780108 726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap starting.
I20250903 14:22:59.785008 726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Neither blocks nor log segments found. Creating new log.
I20250903 14:22:59.786707 726 log.cc:826] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Log is configured to *not* fsync() on all Append() calls
I20250903 14:22:59.796730 726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: No bootstrap required, opened a new log
I20250903 14:22:59.810279 726 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:22:59.810861 726 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:22:59.811058 726 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e83636ce3ca47f3bfc9f84f6755eec2, State: Initialized, Role: FOLLOWER
I20250903 14:22:59.812009 726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:22:59.824158 893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:22:59.824556 893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:22:59.825932 726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: configured and running, proceeding with master startup.
W20250903 14:22:59.878243 908 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:22:59.878396 908 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:22:59.880573 796 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } attempt: 5
W20250903 14:22:59.883250 680 tablet.cc:2378] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:22:59.879823 660 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: OK, instance_id { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" instance_seqno: 1756909379168551 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20250903 14:22:59.886477 821 tablet.cc:2378] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:22:59.903024 796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap starting.
I20250903 14:22:59.912361 796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Neither blocks nor log segments found. Creating new log.
I20250903 14:22:59.913960 796 log.cc:826] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Log is configured to *not* fsync() on all Append() calls
I20250903 14:22:59.914963 660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:22:59.918193 796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: No bootstrap required, opened a new log
I20250903 14:22:59.923861 796 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:22:59.924203 796 raft_consensus.cc:383] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:22:59.924314 796 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Initialized, Role: FOLLOWER
I20250903 14:22:59.925072 796 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:22:59.926680 911 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:22:59.927085 911 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:22:59.928421 796 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:22:59.974175 729 master_runner.cc:263] No leader master found from master f4c58d929e53460883d05e0f14f20780
I20250903 14:22:59.974341 729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:22:59.982461 926 catalog_manager.cc:1560] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 14:22:59.982650 926 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 14:23:00.039640 911 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:00.039997 911 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:00.042140 911 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:00.043592 842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" is_pre_election: true
I20250903 14:23:00.044245 842 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 0.
I20250903 14:23:00.045019 746 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780; no voters:
I20250903 14:23:00.046074 911 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:00.046267 911 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:00.046375 911 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:00.046424 701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" is_pre_election: true
I20250903 14:23:00.046962 701 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 0.
I20250903 14:23:00.051151 911 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:00.052618 701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
I20250903 14:23:00.053000 701 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:00.056818 701 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 1.
I20250903 14:23:00.057595 748 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780; no voters:
I20250903 14:23:00.064486 842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe08ab62bf4073825bf1b598ad2edb"
I20250903 14:23:00.064771 842 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:00.069039 842 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 1.
I20250903 14:23:00.070225 911 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 1 election: Requested vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:00.070806 911 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:00.071337 911 raft_consensus.cc:695] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 1 LEADER]: Becoming Leader. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Running, Role: LEADER
I20250903 14:23:00.072242 911 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:00.085762 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:00.086350 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:00.100239 930 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:23:00.102828 930 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:23:00.119410 842 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Refusing update from remote peer f4c58d929e53460883d05e0f14f20780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:23:00.120692 928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:00.128801 701 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Refusing update from remote peer f4c58d929e53460883d05e0f14f20780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 14:23:00.130125 928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:00.150426 893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:00.150734 893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:00.155607 933 mvcc.cc:204] Tried to move back new op lower bound from 7196300820955099136 to 7196300820792102912. Current Snapshot: MvccSnapshot[applied={T|T < 7196300820955099136}]
I20250903 14:23:00.158730 931 mvcc.cc:204] Tried to move back new op lower bound from 7196300820955099136 to 7196300820792102912. Current Snapshot: MvccSnapshot[applied={T|T < 7196300820955099136}]
I20250903 14:23:00.160018 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:00.160334 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:00.161243 893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:00.161445 893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:00.172375 930 catalog_manager.cc:1349] Generated new cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:00.172526 930 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:23:00.188511 875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:00.188815 875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:00.191556 911 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:00.192270 911 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:00.215960 875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:00.216254 875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:00.224459 930 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 14:23:00.226542 930 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:23:00.254525 930 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Generated new TSK 0
I20250903 14:23:00.255887 930 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:23:00.257414 930 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 14:23:00.259181 930 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 14:23:00.291352 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 14:23:00.754798 889 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:00.754848 943 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:00.754947 889 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: loading cluster ID for follower catalog manager: success
I20250903 14:23:00.755131 943 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:00.756981 943 leader_election.cc:290] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437)
I20250903 14:23:00.757503 700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" is_pre_election: true
I20250903 14:23:00.757627 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:00.758445 817 leader_election.cc:304] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb; no voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:00.758867 943 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 14:23:00.760902 889 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: acquiring CA information for follower catalog manager: success
I20250903 14:23:00.763727 889 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250903 14:23:00.765857 939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:00.766148 939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:00.766247 939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:00.776996 939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:00.777302 939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250903 14:23:00.788317 939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:00.790061 939 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:00.792001 939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:00.802145 946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:00.802104 945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:00.803043 948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:00.803725 939 server_base.cc:1047] running on GCE node
I20250903 14:23:00.804337 939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:00.805451 939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:00.806691 939 hybrid_clock.cc:648] HybridClock initialized: now 1756909380806619 us; error 54 us; skew 500 ppm
I20250903 14:23:00.807063 939 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:00.810133 939 webserver.cc:480] Webserver started at http://127.0.104.1:37267/ using document root <none> and password file <none>
I20250903 14:23:00.810950 939 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:00.811058 939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:00.811484 939 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:23:00.814143 939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data/instance:
uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
format_stamp: "Formatted at 2025-09-03 14:23:00 on dist-test-slave-rxtz"
I20250903 14:23:00.815029 939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/instance:
uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
format_stamp: "Formatted at 2025-09-03 14:23:00 on dist-test-slave-rxtz"
I20250903 14:23:00.821488 939 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250903 14:23:00.826212 954 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:00.828275 939 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250903 14:23:00.828476 939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal
uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
format_stamp: "Formatted at 2025-09-03 14:23:00 on dist-test-slave-rxtz"
I20250903 14:23:00.828698 939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:00.846997 939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:00.847980 939 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:00.848256 939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:00.849342 939 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:23:00.851136 939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:23:00.851246 939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:00.851356 939 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:23:00.851428 939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:00.881186 908 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:00.881312 908 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: loading cluster ID for follower catalog manager: success
I20250903 14:23:00.886044 908 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: acquiring CA information for follower catalog manager: success
I20250903 14:23:00.889483 908 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:00.891263 939 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39849
I20250903 14:23:00.891826 1066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39849 every 8 connection(s)
I20250903 14:23:00.892799 939 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/data/info.pb
I20250903 14:23:00.898761 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 939
I20250903 14:23:00.899003 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/instance
I20250903 14:23:00.904043 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 14:23:00.919699 1067 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:00.920228 1067 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:00.921458 1067 heartbeater.cc:507] Master 127.0.104.62:37899 requested a full tablet report, sending...
I20250903 14:23:00.921643 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:00.921772 1069 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:00.922736 1069 heartbeater.cc:507] Master 127.0.104.61:40437 requested a full tablet report, sending...
I20250903 14:23:00.924513 761 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:00.924515 691 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:00.926616 761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:42939
I20250903 14:23:00.927918 1068 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:00.928128 1068 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:00.928786 1068 heartbeater.cc:507] Master 127.0.104.60:43143 requested a full tablet report, sending...
I20250903 14:23:00.930662 832 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
W20250903 14:23:01.303908 1077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:01.304244 1077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:01.304383 1077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:01.313035 1077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:01.313277 1077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250903 14:23:01.324054 1077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:01.326018 1077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:01.328172 1077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:01.340150 1083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:01.340077 1085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:01.340139 1082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:01.341636 1077 server_base.cc:1047] running on GCE node
I20250903 14:23:01.342499 1077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:01.343700 1077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:01.344949 1077 hybrid_clock.cc:648] HybridClock initialized: now 1756909381344887 us; error 54 us; skew 500 ppm
I20250903 14:23:01.345302 1077 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:01.347676 1077 webserver.cc:480] Webserver started at http://127.0.104.2:42837/ using document root <none> and password file <none>
I20250903 14:23:01.348474 1077 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:01.348587 1077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:01.348942 1077 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:23:01.351648 1077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data/instance:
uuid: "5d4ed06db54c4b5199e190b837bc53fe"
format_stamp: "Formatted at 2025-09-03 14:23:01 on dist-test-slave-rxtz"
I20250903 14:23:01.352507 1077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/instance:
uuid: "5d4ed06db54c4b5199e190b837bc53fe"
format_stamp: "Formatted at 2025-09-03 14:23:01 on dist-test-slave-rxtz"
I20250903 14:23:01.358482 1077 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250903 14:23:01.362882 1091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:01.364530 1077 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:23:01.364698 1077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal
uuid: "5d4ed06db54c4b5199e190b837bc53fe"
format_stamp: "Formatted at 2025-09-03 14:23:01 on dist-test-slave-rxtz"
I20250903 14:23:01.364915 1077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:01.390836 1077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:01.391819 1077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:01.392107 1077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:01.393571 1077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:23:01.395465 1077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:23:01.395583 1077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250903 14:23:01.395725 1077 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:23:01.395812 1077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:01.429647 1069 heartbeater.cc:499] Master 127.0.104.61:40437 was elected leader, sending a full tablet report...
I20250903 14:23:01.437024 1077 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:46863
I20250903 14:23:01.437151 1203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:46863 every 8 connection(s)
I20250903 14:23:01.438622 1077 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/data/info.pb
I20250903 14:23:01.445933 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 1077
I20250903 14:23:01.446211 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/instance
I20250903 14:23:01.454633 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 14:23:01.466028 1205 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:01.466143 1206 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:01.466538 1206 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:01.466543 1205 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:01.466606 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:01.466804 1204 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:01.468222 1204 heartbeater.cc:507] Master 127.0.104.62:37899 requested a full tablet report, sending...
I20250903 14:23:01.468258 1206 heartbeater.cc:507] Master 127.0.104.61:40437 requested a full tablet report, sending...
I20250903 14:23:01.468307 1205 heartbeater.cc:507] Master 127.0.104.60:43143 requested a full tablet report, sending...
I20250903 14:23:01.470415 691 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:01.470433 761 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:01.470506 832 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:01.471709 761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:44683
I20250903 14:23:01.554070 1215 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:01.554306 1215 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:01.555639 1215 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:01.556195 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:01.556321 842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" is_pre_election: true
I20250903 14:23:01.556950 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:01.557380 1215 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 14:23:01.827471 1216 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:01.827724 1216 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:01.828706 1216 leader_election.cc:290] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437)
I20250903 14:23:01.829305 700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" is_pre_election: true
I20250903 14:23:01.829524 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:01.830313 817 leader_election.cc:304] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb; no voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:01.830833 1216 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250903 14:23:01.839715 1214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:01.840065 1214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:01.840204 1214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:01.849495 1214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:01.849776 1214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250903 14:23:01.865089 1214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:01.866864 1214 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:01.869135 1214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:01.880659 1221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:01.880614 1222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:01.881425 1224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:01.881980 1214 server_base.cc:1047] running on GCE node
I20250903 14:23:01.882627 1214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:01.883710 1214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:01.884938 1214 hybrid_clock.cc:648] HybridClock initialized: now 1756909381884859 us; error 62 us; skew 500 ppm
I20250903 14:23:01.885265 1214 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:01.887694 1214 webserver.cc:480] Webserver started at http://127.0.104.3:45493/ using document root <none> and password file <none>
I20250903 14:23:01.888490 1214 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:01.888582 1214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:01.888991 1214 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 14:23:01.891757 1214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data/instance:
uuid: "b03995cb49e94f108cf8caa7a502eafe"
format_stamp: "Formatted at 2025-09-03 14:23:01 on dist-test-slave-rxtz"
I20250903 14:23:01.892594 1214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/instance:
uuid: "b03995cb49e94f108cf8caa7a502eafe"
format_stamp: "Formatted at 2025-09-03 14:23:01 on dist-test-slave-rxtz"
I20250903 14:23:01.899140 1214 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250903 14:23:01.903332 1230 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:01.905217 1214 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250903 14:23:01.905427 1214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal
uuid: "b03995cb49e94f108cf8caa7a502eafe"
format_stamp: "Formatted at 2025-09-03 14:23:01 on dist-test-slave-rxtz"
I20250903 14:23:01.905681 1214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:01.925189 1214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:01.926118 1214 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:01.926398 1214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:01.927489 1214 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 14:23:01.929940 1214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 14:23:01.930049 1214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:01.930177 1214 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 14:23:01.930262 1214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:01.970942 1214 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:43373
I20250903 14:23:01.971055 1342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:43373 every 8 connection(s)
I20250903 14:23:01.972731 1214 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/data/info.pb
I20250903 14:23:01.975176 1206 heartbeater.cc:499] Master 127.0.104.61:40437 was elected leader, sending a full tablet report...
I20250903 14:23:01.978046 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 1214
I20250903 14:23:01.978312 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/instance
I20250903 14:23:02.011263 1343 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:02.011946 1343 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:02.013278 1343 heartbeater.cc:507] Master 127.0.104.62:37899 requested a full tablet report, sending...
I20250903 14:23:02.011055 1344 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:02.014312 1344 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:02.014828 1344 heartbeater.cc:507] Master 127.0.104.60:43143 requested a full tablet report, sending...
I20250903 14:23:02.015401 691 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:02.016083 832 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:02.019187 1347 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:02.019397 1347 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:02.020005 1347 heartbeater.cc:507] Master 127.0.104.61:40437 requested a full tablet report, sending...
I20250903 14:23:02.020974 761 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:02.021837 761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:37331
I20250903 14:23:02.029320 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 14:23:02.068568 755 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_466c344e6f85402282f897ca4ddb8406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.071625 752 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_6b6dbda5e4a448e6b56eb740d7d99a15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.066279 760 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_219a289c9b7a44af91e47feece52778a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.066614 758 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_293e76fad1c145eb963ab9e87c20ce43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.066922 756 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_1bf6146990bd402ea66c580399392859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.069731 757 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_7760fb29e26f4cf88b69689632e4536b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.071575 753 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_ace62b8a67ba4aac89cb28cf186a6816"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.065997 759 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_4aee0c2e9c0b43938b9dc11b7c6bcd9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.065878 761 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_6955211ac77d4f9cb6e5314ebdebf3ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.069250 754 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_d5a7e44d7d1c4005aa2bc39c855d2418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:02.084830 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu pid 797
I20250903 14:23:02.086735 862 generic_service.cc:196] Checking for leaks (request via RPC)
W20250903 14:23:02.087869 756 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1bf6146990bd402ea66c580399392859 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.090831 755 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_466c344e6f85402282f897ca4ddb8406 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:23:02.257805 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu with pid 797
W20250903 14:23:02.436501 1068 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:02.477175 1205 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:02.480894 746 consensus_peers.cc:489] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 -> Peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Couldn't send request to peer 2bbe08ab62bf4073825bf1b598ad2edb. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250903 14:23:02.520200 1344 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:02.524809 1347 heartbeater.cc:499] Master 127.0.104.61:40437 was elected leader, sending a full tablet report...
I20250903 14:23:02.551851 1397 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:02.552177 1397 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:02.553088 1397 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:02.554050 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
W20250903 14:23:02.555894 676 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:02.556057 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:02.556389 1397 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 14:23:02.581219 756 catalog_manager.cc:2149] created HMS catalog entry for table [id=c3c0e03ececf43e1a7c9c2c40acf188e]
W20250903 14:23:02.584570 757 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7760fb29e26f4cf88b69689632e4536b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.584973 754 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.585249 752 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6b6dbda5e4a448e6b56eb740d7d99a15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.584614 753 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ace62b8a67ba4aac89cb28cf186a6816 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.586356 759 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.584975 758 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_293e76fad1c145eb963ab9e87c20ce43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.587348 760 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_219a289c9b7a44af91e47feece52778a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:02.584560 761 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6955211ac77d4f9cb6e5314ebdebf3ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:23:02.613616 756 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_1bf6146990bd402ea66c580399392859" } new_table_name: "default.Table_31acfe71a94349f187dc3fa60a6ecc89" modify_external_catalogs: true
I20250903 14:23:02.630599 755 catalog_manager.cc:2149] created HMS catalog entry for table [id=2224cdb559fe45599b2464369cd2b483]
I20250903 14:23:02.654382 1002 tablet_service.cc:1468] Processing CreateTablet for tablet 819c70f5de39472386cb2c27543ed8fa (DEFAULT_TABLE table=default.table_1bf6146990bd402ea66c580399392859 [id=c3c0e03ececf43e1a7c9c2c40acf188e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.658001 1001 tablet_service.cc:1468] Processing CreateTablet for tablet d795310ddb1244ef9fd753936b345b1b (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.659000 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d795310ddb1244ef9fd753936b345b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.661633 755 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_466c344e6f85402282f897ca4ddb8406" } new_table_name: "default.Table_3368c82b8e6b4b149e8ad8d925c3d0e0" modify_external_catalogs: true
I20250903 14:23:02.667999 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819c70f5de39472386cb2c27543ed8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.672000 1413 tablet_bootstrap.cc:492] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:02.688469 1413 tablet_bootstrap.cc:654] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.690119 1413 log.cc:826] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:02.695327 757 catalog_manager.cc:2149] created HMS catalog entry for table [id=69ed0433e40249699066ac4a0b3825a1]
I20250903 14:23:02.696393 1139 tablet_service.cc:1468] Processing CreateTablet for tablet 819c70f5de39472386cb2c27543ed8fa (DEFAULT_TABLE table=default.table_1bf6146990bd402ea66c580399392859 [id=c3c0e03ececf43e1a7c9c2c40acf188e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.698316 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819c70f5de39472386cb2c27543ed8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.699803 1138 tablet_service.cc:1468] Processing CreateTablet for tablet d795310ddb1244ef9fd753936b345b1b (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.703810 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d795310ddb1244ef9fd753936b345b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.705677 757 master_service.cc:948] ReplaceTablet: received request to replace tablet d795310ddb1244ef9fd753936b345b1b from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:02.708463 1277 tablet_service.cc:1468] Processing CreateTablet for tablet d795310ddb1244ef9fd753936b345b1b (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.710403 1277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d795310ddb1244ef9fd753936b345b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.711534 1278 tablet_service.cc:1468] Processing CreateTablet for tablet 819c70f5de39472386cb2c27543ed8fa (DEFAULT_TABLE table=default.table_1bf6146990bd402ea66c580399392859 [id=c3c0e03ececf43e1a7c9c2c40acf188e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.712188 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819c70f5de39472386cb2c27543ed8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.720268 1413 tablet_bootstrap.cc:492] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:02.720628 1413 ts_tablet_manager.cc:1397] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.000s
I20250903 14:23:02.727996 1413 raft_consensus.cc:357] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.728387 1413 raft_consensus.cc:383] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.728508 1413 raft_consensus.cc:738] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.729234 1413 consensus_queue.cc:260] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.738026 1413 ts_tablet_manager.cc:1428] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20250903 14:23:02.740924 1413 tablet_bootstrap.cc:492] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:02.743099 1413 tablet_bootstrap.cc:654] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.746959 1413 tablet_bootstrap.cc:492] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:02.747224 1413 ts_tablet_manager.cc:1397] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250903 14:23:02.748289 1413 raft_consensus.cc:357] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.748533 1413 raft_consensus.cc:383] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.748623 1413 raft_consensus.cc:738] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.749027 1413 consensus_queue.cc:260] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.752951 1413 ts_tablet_manager.cc:1428] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250903 14:23:02.756788 1418 tablet_bootstrap.cc:492] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:02.760113 757 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Sending DeleteTablet for 3 replicas of tablet d795310ddb1244ef9fd753936b345b1b
I20250903 14:23:02.760929 757 catalog_manager.cc:6652] ReplaceTablet: tablet d795310ddb1244ef9fd753936b345b1b deleted and replaced by tablet 574c4a8c94d04d4ba30965276b87a97e
I20250903 14:23:02.767175 752 catalog_manager.cc:2149] created HMS catalog entry for table [id=8590620c66374843adbe167d46c1de13]
I20250903 14:23:02.767997 757 master_service.cc:948] ReplaceTablet: received request to replace tablet d795310ddb1244ef9fd753936b345b1b from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:02.767937 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet d795310ddb1244ef9fd753936b345b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c4a8c94d04d4ba30965276b87a97e) from {username='slave'} at 127.0.0.1:55526
I20250903 14:23:02.768260 1276 tablet_service.cc:1515] Processing DeleteTablet for tablet d795310ddb1244ef9fd753936b345b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c4a8c94d04d4ba30965276b87a97e) from {username='slave'} at 127.0.0.1:58704
I20250903 14:23:02.772606 1419 tablet_bootstrap.cc:492] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:02.776188 1275 tablet_service.cc:1468] Processing CreateTablet for tablet c51713a4e7554285ba265825b8651efe (DEFAULT_TABLE table=default.table_7760fb29e26f4cf88b69689632e4536b [id=69ed0433e40249699066ac4a0b3825a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.776924 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51713a4e7554285ba265825b8651efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.777344 1138 tablet_service.cc:1515] Processing DeleteTablet for tablet d795310ddb1244ef9fd753936b345b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c4a8c94d04d4ba30965276b87a97e) from {username='slave'} at 127.0.0.1:42548
I20250903 14:23:02.777465 1419 tablet_bootstrap.cc:654] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.782471 1420 tablet_replica.cc:331] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
W20250903 14:23:02.782508 748 catalog_manager.cc:4908] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): delete failed for tablet d795310ddb1244ef9fd753936b345b1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d795310ddb1244ef9fd753936b345b1b already in progress: creating tablet
I20250903 14:23:02.783182 1420 raft_consensus.cc:2241] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:02.784669 752 master_service.cc:948] ReplaceTablet: received request to replace tablet d795310ddb1244ef9fd753936b345b1b from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:02.785223 752 master_service.cc:948] ReplaceTablet: received request to replace tablet d795310ddb1244ef9fd753936b345b1b from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:02.785622 752 master_service.cc:948] ReplaceTablet: received request to replace tablet d795310ddb1244ef9fd753936b345b1b from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:02.786111 752 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_7760fb29e26f4cf88b69689632e4536b" } new_table_name: "default.Table_2e6fe8bc133e4eb68510d15bd758709a" modify_external_catalogs: true
I20250903 14:23:02.787895 1139 tablet_service.cc:1468] Processing CreateTablet for tablet c51713a4e7554285ba265825b8651efe (DEFAULT_TABLE table=default.table_7760fb29e26f4cf88b69689632e4536b [id=69ed0433e40249699066ac4a0b3825a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.788388 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51713a4e7554285ba265825b8651efe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 14:23:02.789979 748 catalog_manager.cc:4908] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): delete failed for tablet d795310ddb1244ef9fd753936b345b1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d795310ddb1244ef9fd753936b345b1b already in progress: creating tablet
I20250903 14:23:02.791256 1420 raft_consensus.cc:2270] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:02.794394 757 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_6b6dbda5e4a448e6b56eb740d7d99a15" } new_table_name: "default.Table_bd0820d81cd1476c9b6474c12f0053f6" modify_external_catalogs: true
I20250903 14:23:02.795259 1420 ts_tablet_manager.cc:1905] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:02.797605 1419 log.cc:826] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:02.797919 1001 tablet_service.cc:1468] Processing CreateTablet for tablet 574c4a8c94d04d4ba30965276b87a97e (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.798364 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574c4a8c94d04d4ba30965276b87a97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.800570 1002 tablet_service.cc:1468] Processing CreateTablet for tablet c51713a4e7554285ba265825b8651efe (DEFAULT_TABLE table=default.table_7760fb29e26f4cf88b69689632e4536b [id=69ed0433e40249699066ac4a0b3825a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.801098 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51713a4e7554285ba265825b8651efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.801271 1418 tablet_bootstrap.cc:654] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.802862 1418 log.cc:826] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:02.803828 1139 tablet_service.cc:1468] Processing CreateTablet for tablet 574c4a8c94d04d4ba30965276b87a97e (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.804267 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574c4a8c94d04d4ba30965276b87a97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.804487 1275 tablet_service.cc:1468] Processing CreateTablet for tablet 574c4a8c94d04d4ba30965276b87a97e (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.806147 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574c4a8c94d04d4ba30965276b87a97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.812902 1137 tablet_service.cc:1468] Processing CreateTablet for tablet 37f6ddf1daea4a228072253412267924 (DEFAULT_TABLE table=default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.813555 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f6ddf1daea4a228072253412267924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.815896 1413 tablet_bootstrap.cc:492] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:02.816898 1420 ts_tablet_manager.cc:1918] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:02.817098 1420 log.cc:1199] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/d795310ddb1244ef9fd753936b345b1b
I20250903 14:23:02.817586 1420 ts_tablet_manager.cc:1939] T d795310ddb1244ef9fd753936b345b1b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:02.817859 1413 tablet_bootstrap.cc:654] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.819859 748 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet d795310ddb1244ef9fd753936b345b1b (table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]) successfully deleted
I20250903 14:23:02.823828 1419 tablet_bootstrap.cc:492] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:02.824213 1419 ts_tablet_manager.cc:1397] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.004s
I20250903 14:23:02.825587 1001 tablet_service.cc:1468] Processing CreateTablet for tablet 37f6ddf1daea4a228072253412267924 (DEFAULT_TABLE table=default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.826107 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f6ddf1daea4a228072253412267924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.826870 1418 tablet_bootstrap.cc:492] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:02.827193 1418 ts_tablet_manager.cc:1397] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.006s
I20250903 14:23:02.828387 1413 tablet_bootstrap.cc:492] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:02.828547 1413 ts_tablet_manager.cc:1397] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.007s
I20250903 14:23:02.828698 1278 tablet_service.cc:1468] Processing CreateTablet for tablet 37f6ddf1daea4a228072253412267924 (DEFAULT_TABLE table=default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.829183 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f6ddf1daea4a228072253412267924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.829427 1413 raft_consensus.cc:357] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:02.829612 1413 raft_consensus.cc:383] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.829704 1413 raft_consensus.cc:738] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.829937 1413 consensus_queue.cc:260] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:02.832912 1413 ts_tablet_manager.cc:1428] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250903 14:23:02.837810 1413 tablet_bootstrap.cc:492] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:02.838469 1419 raft_consensus.cc:357] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.838955 1419 raft_consensus.cc:383] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.839102 1419 raft_consensus.cc:738] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.839679 1413 tablet_bootstrap.cc:654] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.839987 1419 consensus_queue.cc:260] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.846143 1418 raft_consensus.cc:357] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.846635 1418 raft_consensus.cc:383] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.846787 1418 raft_consensus.cc:738] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.852499 1418 consensus_queue.cc:260] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.859002 1418 ts_tablet_manager.cc:1428] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.032s user 0.010s sys 0.003s
I20250903 14:23:02.860360 753 catalog_manager.cc:2149] created HMS catalog entry for table [id=d237316ba9fd4366a139c31dd9524328]
I20250903 14:23:02.863155 1418 tablet_bootstrap.cc:492] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:02.865406 1419 ts_tablet_manager.cc:1428] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.041s user 0.014s sys 0.000s
I20250903 14:23:02.866181 1419 tablet_bootstrap.cc:492] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:02.868208 1419 tablet_bootstrap.cc:654] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.868574 1413 tablet_bootstrap.cc:492] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:02.868749 1413 ts_tablet_manager.cc:1397] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.012s
I20250903 14:23:02.870882 1416 raft_consensus.cc:491] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:02.871191 1416 raft_consensus.cc:513] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.883267 1138 tablet_service.cc:1515] Processing DeleteTablet for tablet d795310ddb1244ef9fd753936b345b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c4a8c94d04d4ba30965276b87a97e) from {username='slave'} at 127.0.0.1:42548
I20250903 14:23:02.883801 1422 tablet_replica.cc:331] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:02.884508 1422 raft_consensus.cc:2241] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:02.885063 1422 raft_consensus.cc:2270] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:02.885443 1416 leader_election.cc:290] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:02.890338 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819c70f5de39472386cb2c27543ed8fa" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:02.893370 1413 raft_consensus.cc:357] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:02.893675 1413 raft_consensus.cc:383] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.893765 1413 raft_consensus.cc:738] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.894061 1413 consensus_queue.cc:260] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
W20250903 14:23:02.897130 1070 tablet.cc:2378] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:23:02.900831 1422 ts_tablet_manager.cc:1905] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:02.901226 1001 tablet_service.cc:1468] Processing CreateTablet for tablet b09ba744dc744fc3ac61383549c4babc (DEFAULT_TABLE table=default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.901679 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09ba744dc744fc3ac61383549c4babc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.905570 1275 tablet_service.cc:1468] Processing CreateTablet for tablet b09ba744dc744fc3ac61383549c4babc (DEFAULT_TABLE table=default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.906091 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09ba744dc744fc3ac61383549c4babc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.907797 1138 tablet_service.cc:1468] Processing CreateTablet for tablet b09ba744dc744fc3ac61383549c4babc (DEFAULT_TABLE table=default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:02.908445 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09ba744dc744fc3ac61383549c4babc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:02.917596 1413 ts_tablet_manager.cc:1428] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.049s user 0.001s sys 0.002s
I20250903 14:23:02.918103 1413 tablet_bootstrap.cc:492] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:02.920544 1413 tablet_bootstrap.cc:654] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.920894 1422 ts_tablet_manager.cc:1918] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:02.921087 1422 log.cc:1199] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/d795310ddb1244ef9fd753936b345b1b
I20250903 14:23:02.921598 1422 ts_tablet_manager.cc:1939] T d795310ddb1244ef9fd753936b345b1b P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:02.924978 1418 tablet_bootstrap.cc:654] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.927152 1419 tablet_bootstrap.cc:492] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:02.927415 1419 ts_tablet_manager.cc:1397] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.032s
I20250903 14:23:02.928854 1419 raft_consensus.cc:357] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.929193 1419 raft_consensus.cc:383] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.929282 1419 raft_consensus.cc:738] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.929647 1419 consensus_queue.cc:260] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.930245 1419 ts_tablet_manager.cc:1428] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250903 14:23:02.933621 753 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_867ae72d30fd4ea5b2e85874f80942ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:23:02.933701 958 leader_election.cc:343] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Illegal state: must be running to vote when last-logged opid is not known
I20250903 14:23:02.933868 1413 tablet_bootstrap.cc:492] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:02.934024 1413 ts_tablet_manager.cc:1397] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250903 14:23:02.934973 1413 raft_consensus.cc:357] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:02.935175 1413 raft_consensus.cc:383] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.935258 1413 raft_consensus.cc:738] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.935518 1413 consensus_queue.cc:260] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:02.923679 748 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet d795310ddb1244ef9fd753936b345b1b (table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]) successfully deleted
I20250903 14:23:02.938359 1413 ts_tablet_manager.cc:1428] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 14:23:02.939229 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819c70f5de39472386cb2c27543ed8fa" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:02.939810 1298 raft_consensus.cc:2466] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:02.940588 958 leader_election.cc:304] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters: 5d4ed06db54c4b5199e190b837bc53fe
I20250903 14:23:02.941107 1416 raft_consensus.cc:2802] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:02.941251 1416 raft_consensus.cc:491] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:02.941358 1416 raft_consensus.cc:3058] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:02.942324 1419 tablet_bootstrap.cc:492] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:02.944514 1419 tablet_bootstrap.cc:654] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.944613 1416 raft_consensus.cc:513] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.945343 1416 leader_election.cc:290] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:02.947929 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819c70f5de39472386cb2c27543ed8fa" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:02.948908 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819c70f5de39472386cb2c27543ed8fa" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:02.949194 1298 raft_consensus.cc:3058] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:02.952868 1298 raft_consensus.cc:2466] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:02.953320 1413 tablet_bootstrap.cc:492] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:02.955152 958 leader_election.cc:304] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:02.955726 1416 raft_consensus.cc:2802] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:02.956202 1418 tablet_bootstrap.cc:492] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:02.956393 1418 ts_tablet_manager.cc:1397] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.000s
W20250903 14:23:02.956640 958 leader_election.cc:343] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Illegal state: must be running to vote when last-logged opid is not known
I20250903 14:23:02.957185 1418 raft_consensus.cc:357] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.957397 1418 raft_consensus.cc:383] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.957468 1418 raft_consensus.cc:738] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.957712 1418 consensus_queue.cc:260] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.958190 1418 ts_tablet_manager.cc:1428] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:23:02.958532 1418 tablet_bootstrap.cc:492] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:02.955325 1413 tablet_bootstrap.cc:654] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.962544 1418 tablet_bootstrap.cc:654] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.965565 1416 raft_consensus.cc:695] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:02.965981 759 catalog_manager.cc:2149] created HMS catalog entry for table [id=829c730ca4ee4d2792c66bf244615417]
I20250903 14:23:02.966656 1416 consensus_queue.cc:237] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:02.972323 1419 tablet_bootstrap.cc:492] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:02.972523 1419 ts_tablet_manager.cc:1397] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.007s
I20250903 14:23:02.973510 1419 raft_consensus.cc:357] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:02.973713 1419 raft_consensus.cc:383] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:02.973831 1419 raft_consensus.cc:738] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:02.974148 1419 consensus_queue.cc:260] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:02.975966 1419 ts_tablet_manager.cc:1428] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 14:23:02.977185 1419 tablet_bootstrap.cc:492] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:02.977528 1404 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:02.977830 1404 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:02.979122 1419 tablet_bootstrap.cc:654] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:02.994347 1275 tablet_service.cc:1515] Processing DeleteTablet for tablet d795310ddb1244ef9fd753936b345b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c4a8c94d04d4ba30965276b87a97e) from {username='slave'} at 127.0.0.1:58704
I20250903 14:23:02.994737 1421 tablet_replica.cc:331] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:02.995322 1421 raft_consensus.cc:2241] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:02.995936 1421 raft_consensus.cc:2270] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:02.999640 1421 ts_tablet_manager.cc:1905] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:03.001214 1138 tablet_service.cc:1468] Processing CreateTablet for tablet cbd068ad8e4448ba96e1bef32449a7c0 (DEFAULT_TABLE table=default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.001253 1001 tablet_service.cc:1468] Processing CreateTablet for tablet cbd068ad8e4448ba96e1bef32449a7c0 (DEFAULT_TABLE table=default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.001677 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd068ad8e4448ba96e1bef32449a7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.001691 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd068ad8e4448ba96e1bef32449a7c0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 14:23:03.005574 1351 tablet.cc:2378] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:23:03.005910 1275 tablet_service.cc:1468] Processing CreateTablet for tablet cbd068ad8e4448ba96e1bef32449a7c0 (DEFAULT_TABLE table=default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.006409 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd068ad8e4448ba96e1bef32449a7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.009184 1421 ts_tablet_manager.cc:1918] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:03.012532 1413 tablet_bootstrap.cc:492] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.012715 1413 ts_tablet_manager.cc:1397] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.001s
I20250903 14:23:03.013578 1413 raft_consensus.cc:357] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.013796 1413 raft_consensus.cc:383] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.013880 1413 raft_consensus.cc:738] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.014166 1413 consensus_queue.cc:260] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.014685 1413 ts_tablet_manager.cc:1428] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 14:23:03.015098 1413 tablet_bootstrap.cc:492] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:03.012086 1419 tablet_bootstrap.cc:492] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.016960 1413 tablet_bootstrap.cc:654] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.017083 1419 ts_tablet_manager.cc:1397] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.016s
I20250903 14:23:03.018039 1419 raft_consensus.cc:357] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.018340 1419 raft_consensus.cc:383] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.018462 1419 raft_consensus.cc:738] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.018779 1419 consensus_queue.cc:260] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.019625 1421 log.cc:1199] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/d795310ddb1244ef9fd753936b345b1b
I20250903 14:23:03.024716 1421 ts_tablet_manager.cc:1939] T d795310ddb1244ef9fd753936b345b1b P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:03.026858 748 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet d795310ddb1244ef9fd753936b345b1b (table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]) successfully deleted
I20250903 14:23:03.030958 1278 tablet_service.cc:1515] Processing DeleteTablet for tablet d795310ddb1244ef9fd753936b345b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c4a8c94d04d4ba30965276b87a97e) from {username='slave'} at 127.0.0.1:58704
W20250903 14:23:03.031752 748 catalog_manager.cc:4895] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): delete failed for tablet d795310ddb1244ef9fd753936b345b1b because the tablet was not found. No further retry: Not found: Tablet not found: d795310ddb1244ef9fd753936b345b1b
I20250903 14:23:03.032536 1418 tablet_bootstrap.cc:492] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.032729 1418 ts_tablet_manager.cc:1397] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.007s
I20250903 14:23:03.033418 759 catalog_manager.cc:5582] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:03.033881 1418 raft_consensus.cc:357] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.034063 1418 raft_consensus.cc:383] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.034176 1418 raft_consensus.cc:738] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.019451 1419 ts_tablet_manager.cc:1428] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 14:23:03.036576 1419 tablet_bootstrap.cc:492] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.038151 1418 consensus_queue.cc:260] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.038707 1419 tablet_bootstrap.cc:654] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.041939 1418 ts_tablet_manager.cc:1428] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20250903 14:23:03.044375 1418 tablet_bootstrap.cc:492] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.046847 1418 tablet_bootstrap.cc:654] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.055068 1413 tablet_bootstrap.cc:492] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.055274 1413 ts_tablet_manager.cc:1397] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.001s
I20250903 14:23:03.056257 1413 raft_consensus.cc:357] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.056460 1413 raft_consensus.cc:383] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.056558 1413 raft_consensus.cc:738] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.056787 1413 consensus_queue.cc:260] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.057482 1413 ts_tablet_manager.cc:1428] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250903 14:23:03.062925 758 catalog_manager.cc:2149] created HMS catalog entry for table [id=671393d4fe2249da97a8068fce39e09d]
I20250903 14:23:03.064078 759 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_eb580089aaa843ff90baf8a5795d8f56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:03.068219 1419 tablet_bootstrap.cc:492] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.068531 1419 ts_tablet_manager.cc:1397] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20250903 14:23:03.069147 758 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 4) took 1003 ms. Trace:
I20250903 14:23:03.069376 1418 tablet_bootstrap.cc:492] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.069562 1418 ts_tablet_manager.cc:1397] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20250903 14:23:03.069496 1419 raft_consensus.cc:357] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.069664 1419 raft_consensus.cc:383] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.069747 1419 raft_consensus.cc:738] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.070096 1419 consensus_queue.cc:260] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.070331 1427 raft_consensus.cc:491] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.070495 1427 raft_consensus.cc:513] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.069303 758 rpcz_store.cc:276] 0903 14:23:02.065580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.066395 (+ 815us) service_pool.cc:225] Handling call
0903 14:23:02.587099 (+520704us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:02.587596 (+ 497us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:03.062848 (+475252us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:03.068854 (+ 6006us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:03.068988 (+ 134us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:03.069094 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:03.063835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:03.063997 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:03.064004 (+ 7us) write_op.cc:435] Acquired schema lock
0903 14:23:03.064006 (+ 2us) tablet.cc:660] Decoding operations
0903 14:23:03.064113 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:03.064130 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:03.064134 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:03.064238 (+ 104us) tablet.cc:699] Row locks acquired
0903 14:23:03.064240 (+ 2us) write_op.cc:260] PREPARE: finished
0903 14:23:03.064286 (+ 46us) write_op.cc:270] Start()
0903 14:23:03.064326 (+ 40us) write_op.cc:276] Timestamp: P: 1756909383064227 usec, L: 0
0903 14:23:03.064330 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 14:23:03.064525 (+ 195us) log.cc:844] Serialized 432 byte log entry
0903 14:23:03.068031 (+ 3506us) op_driver.cc:464] REPLICATION: finished
0903 14:23:03.068217 (+ 186us) write_op.cc:301] APPLY: starting
0903 14:23:03.068236 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:03.068297 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:03.068321 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:03.068423 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:03.068458 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:03.068465 (+ 7us) write_op.cc:312] APPLY: finished
0903 14:23:03.068554 (+ 89us) log.cc:844] Serialized 28 byte log entry
0903 14:23:03.068630 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.068651 (+ 21us) write_op.cc:454] Released schema lock
0903 14:23:03.068657 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:03.068731 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.068734 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":377971,"Hive Metastore.run_cpu_time_us":291,"Hive Metastore.run_wall_time_us":96473,"spinlock_wait_cycles":14848,"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":507,"apply.run_wall_time_us":507,"num_ops":2,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":821,"raft.queue_time_us":498,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":238,"replication_time_us":3670}]]}
I20250903 14:23:03.072634 1427 leader_election.cc:290] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.073508 1418 raft_consensus.cc:357] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.073736 1418 raft_consensus.cc:383] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.073832 1418 raft_consensus.cc:738] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.074083 1418 consensus_queue.cc:260] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.075985 1418 ts_tablet_manager.cc:1428] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250903 14:23:03.085148 1418 tablet_bootstrap.cc:492] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.087007 1418 tablet_bootstrap.cc:654] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.090479 1419 ts_tablet_manager.cc:1428] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.022s user 0.010s sys 0.010s
I20250903 14:23:03.091742 1418 tablet_bootstrap.cc:492] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.091938 1418 ts_tablet_manager.cc:1397] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250903 14:23:03.090883 1419 tablet_bootstrap.cc:492] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.092800 1418 raft_consensus.cc:357] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.093019 1418 raft_consensus.cc:383] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.093133 1418 raft_consensus.cc:738] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.093398 1418 consensus_queue.cc:260] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.094103 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574c4a8c94d04d4ba30965276b87a97e" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:03.094451 1159 raft_consensus.cc:2466] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.095152 1234 leader_election.cc:304] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.095408 1419 tablet_bootstrap.cc:654] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.095993 1418 ts_tablet_manager.cc:1428] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 14:23:03.097299 1418 tablet_bootstrap.cc:492] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.099298 1418 tablet_bootstrap.cc:654] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.099519 1001 tablet_service.cc:1468] Processing CreateTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f (DEFAULT_TABLE table=default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.100133 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc5829b0f9d4facb2d6da58c6be842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.100440 1443 raft_consensus.cc:2802] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.100574 1443 raft_consensus.cc:491] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.100709 1443 raft_consensus.cc:3058] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.104111 1443 raft_consensus.cc:513] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.104912 1443 leader_election.cc:290] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.106479 1138 tablet_service.cc:1468] Processing CreateTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f (DEFAULT_TABLE table=default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.106967 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc5829b0f9d4facb2d6da58c6be842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.108429 1275 tablet_service.cc:1468] Processing CreateTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f (DEFAULT_TABLE table=default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.109009 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc5829b0f9d4facb2d6da58c6be842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.112160 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574c4a8c94d04d4ba30965276b87a97e" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:03.112483 1159 raft_consensus.cc:3058] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.114701 1418 tablet_bootstrap.cc:492] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.114923 1418 ts_tablet_manager.cc:1397] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250903 14:23:03.115775 1418 raft_consensus.cc:357] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.116037 1418 raft_consensus.cc:383] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.116125 1418 raft_consensus.cc:738] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.116382 1418 consensus_queue.cc:260] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.118132 1418 ts_tablet_manager.cc:1428] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:03.119648 1418 tablet_bootstrap.cc:492] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.121598 1418 tablet_bootstrap.cc:654] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.130126 1418 tablet_bootstrap.cc:492] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.130324 1418 ts_tablet_manager.cc:1397] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20250903 14:23:03.131228 1418 raft_consensus.cc:357] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.131495 1418 raft_consensus.cc:383] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.131608 1418 raft_consensus.cc:738] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.131897 1418 consensus_queue.cc:260] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.133700 1418 ts_tablet_manager.cc:1428] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:03.135934 1418 tablet_bootstrap.cc:492] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.137377 1159 raft_consensus.cc:2466] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.138032 1418 tablet_bootstrap.cc:654] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.140421 1418 tablet_bootstrap.cc:492] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.140625 1418 ts_tablet_manager.cc:1397] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:23:03.141026 1234 leader_election.cc:304] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.141479 1443 raft_consensus.cc:2802] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.141746 1443 raft_consensus.cc:695] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Becoming Leader. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Running, Role: LEADER
I20250903 14:23:03.142379 1443 consensus_queue.cc:237] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.145776 1418 raft_consensus.cc:357] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.146046 1418 raft_consensus.cc:383] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.146132 1418 raft_consensus.cc:738] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.146471 1418 consensus_queue.cc:260] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.147708 1418 ts_tablet_manager.cc:1428] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250903 14:23:03.151821 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574c4a8c94d04d4ba30965276b87a97e" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.152297 1022 raft_consensus.cc:2466] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.151799 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574c4a8c94d04d4ba30965276b87a97e" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.152849 1021 raft_consensus.cc:3058] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.155762 754 catalog_manager.cc:2149] created HMS catalog entry for table [id=d35e7f78eeb345df9ad80071d3b6ccb9]
I20250903 14:23:03.157418 1021 raft_consensus.cc:2466] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.162729 1419 tablet_bootstrap.cc:492] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.162961 1419 ts_tablet_manager.cc:1397] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.072s user 0.027s sys 0.039s
I20250903 14:23:03.163858 1419 raft_consensus.cc:357] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.164094 1419 raft_consensus.cc:383] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.164176 1419 raft_consensus.cc:738] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.164458 1419 consensus_queue.cc:260] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.164983 1419 ts_tablet_manager.cc:1428] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:23:03.165328 1419 tablet_bootstrap.cc:492] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.167116 1419 tablet_bootstrap.cc:654] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.170722 1419 tablet_bootstrap.cc:492] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.170949 1419 ts_tablet_manager.cc:1397] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250903 14:23:03.171990 1419 raft_consensus.cc:357] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.173800 1419 raft_consensus.cc:383] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.173954 1419 raft_consensus.cc:738] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.174336 1419 consensus_queue.cc:260] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.176802 1419 ts_tablet_manager.cc:1428] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 14:23:03.178076 1419 tablet_bootstrap.cc:492] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.178613 1413 tablet_bootstrap.cc:492] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:03.180070 1419 tablet_bootstrap.cc:654] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.180533 1413 tablet_bootstrap.cc:654] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.184574 754 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 8) took 1116 ms. Trace:
I20250903 14:23:03.184736 754 rpcz_store.cc:276] 0903 14:23:02.068377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.068436 (+ 59us) service_pool.cc:225] Handling call
0903 14:23:02.585153 (+516717us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:02.585845 (+ 692us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:03.155686 (+569841us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:03.180765 (+ 25079us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:03.184382 (+ 3617us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:03.184510 (+ 128us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:03.156747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:03.156922 (+ 175us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:03.156933 (+ 11us) write_op.cc:435] Acquired schema lock
0903 14:23:03.156938 (+ 5us) tablet.cc:660] Decoding operations
0903 14:23:03.157094 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:03.157121 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:03.157131 (+ 10us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:03.157249 (+ 118us) tablet.cc:699] Row locks acquired
0903 14:23:03.157253 (+ 4us) write_op.cc:260] PREPARE: finished
0903 14:23:03.157303 (+ 50us) write_op.cc:270] Start()
0903 14:23:03.157383 (+ 80us) write_op.cc:276] Timestamp: P: 1756909383157232 usec, L: 0
0903 14:23:03.157389 (+ 6us) op_driver.cc:348] REPLICATION: starting
0903 14:23:03.170253 (+ 12864us) log.cc:844] Serialized 432 byte log entry
0903 14:23:03.170691 (+ 438us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x616000050e54. stack: 00007f995341bdc6 00007f995341bb80 00007f99524033ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f995ee7ad62 00007f99602232d5 00007f9960221df9 00007f996022a220 00007f996022a02d 0000000000a774dc 00007f995346214f 00007f995346541b 00007f9953465298 0000000000a774dc
0903 14:23:03.179999 (+ 9308us) op_driver.cc:464] REPLICATION: finished
0903 14:23:03.180196 (+ 197us) write_op.cc:301] APPLY: starting
0903 14:23:03.180215 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:03.180265 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:03.180268 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:03.180393 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:03.180441 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:03.180450 (+ 9us) write_op.cc:312] APPLY: finished
0903 14:23:03.180548 (+ 98us) log.cc:844] Serialized 28 byte log entry
0903 14:23:03.180624 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.180649 (+ 25us) write_op.cc:454] Released schema lock
0903 14:23:03.180658 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:03.180750 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.180753 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":476212,"Hive Metastore.run_cpu_time_us":376,"Hive Metastore.run_wall_time_us":93134,"mutex_wait_us":1725,"spinlock_wait_cycles":9856,"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":546,"apply.run_wall_time_us":545,"num_ops":2,"prepare.queue_time_us":71,"prepare.run_cpu_time_us":1632,"prepare.run_wall_time_us":14043,"raft.queue_time_us":957,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":369,"replication_time_us":22571,"spinlock_wait_cycles":28921728}]]}
I20250903 14:23:03.185628 754 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_ace62b8a67ba4aac89cb28cf186a6816" } new_table_name: "default.Table_d7a178c8bc1546dd9464cf497775d66a" modify_external_catalogs: true
I20250903 14:23:03.185451 758 catalog_manager.cc:5582] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to b03995cb49e94f108cf8caa7a502eafe (127.0.104.3). New cstate: current_term: 1 leader_uuid: "b03995cb49e94f108cf8caa7a502eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:03.188802 1413 tablet_bootstrap.cc:492] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.189018 1413 ts_tablet_manager.cc:1397] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20250903 14:23:03.190057 1413 raft_consensus.cc:357] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.190282 1413 raft_consensus.cc:383] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.190352 1413 raft_consensus.cc:738] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.190598 1413 consensus_queue.cc:260] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.191092 1413 ts_tablet_manager.cc:1428] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 14:23:03.193506 1443 raft_consensus.cc:491] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.193688 1443 raft_consensus.cc:513] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.194409 1443 leader_election.cc:290] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.195200 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c51713a4e7554285ba265825b8651efe" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.195572 1021 raft_consensus.cc:2466] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.196139 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c51713a4e7554285ba265825b8651efe" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:03.196381 1159 raft_consensus.cc:2466] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.196750 1234 leader_election.cc:304] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.199224 1443 raft_consensus.cc:2802] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.199368 1443 raft_consensus.cc:491] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.199472 1443 raft_consensus.cc:3058] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.200423 1419 tablet_bootstrap.cc:492] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.200609 1419 ts_tablet_manager.cc:1397] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.003s
I20250903 14:23:03.201304 1419 raft_consensus.cc:357] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.201470 1419 raft_consensus.cc:383] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.201551 1419 raft_consensus.cc:738] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.201792 1419 consensus_queue.cc:260] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.202279 1419 ts_tablet_manager.cc:1428] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 14:23:03.202433 1443 raft_consensus.cc:513] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
W20250903 14:23:03.203322 1207 tablet.cc:2378] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 14:23:03.203483 1428 raft_consensus.cc:491] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.203663 1428 raft_consensus.cc:513] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.204619 1443 leader_election.cc:290] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.205586 1428 leader_election.cc:290] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.206485 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c51713a4e7554285ba265825b8651efe" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:03.206693 1159 raft_consensus.cc:3058] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.209334 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c51713a4e7554285ba265825b8651efe" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.209514 1159 raft_consensus.cc:2466] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.209581 1021 raft_consensus.cc:3058] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.212666 1021 raft_consensus.cc:2466] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.213575 1234 leader_election.cc:304] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.214249 1443 raft_consensus.cc:2802] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.214458 1443 raft_consensus.cc:695] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Becoming Leader. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Running, Role: LEADER
I20250903 14:23:03.214811 1443 consensus_queue.cc:237] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.222075 1275 tablet_service.cc:1468] Processing CreateTablet for tablet b5afc9676cca4a9ba1841546f5d38d7a (DEFAULT_TABLE table=default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.222584 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5afc9676cca4a9ba1841546f5d38d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.223107 1138 tablet_service.cc:1468] Processing CreateTablet for tablet b5afc9676cca4a9ba1841546f5d38d7a (DEFAULT_TABLE table=default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.222478 1001 tablet_service.cc:1468] Processing CreateTablet for tablet b5afc9676cca4a9ba1841546f5d38d7a (DEFAULT_TABLE table=default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.223778 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5afc9676cca4a9ba1841546f5d38d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.223821 760 catalog_manager.cc:2149] created HMS catalog entry for table [id=03d2198ad57f4f729123cf099287a61d]
I20250903 14:23:03.225917 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5afc9676cca4a9ba1841546f5d38d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.229171 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6ddf1daea4a228072253412267924" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:03.229444 1298 raft_consensus.cc:2466] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.230218 1095 leader_election.cc:304] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.230708 1428 raft_consensus.cc:2802] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.230829 1428 raft_consensus.cc:491] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.230926 1428 raft_consensus.cc:3058] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.234292 1428 raft_consensus.cc:513] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.235154 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43143
--webserver_interface=127.0.104.60
--webserver_port=41549
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 14:23:03.237358 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6ddf1daea4a228072253412267924" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.237461 760 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 2) took 1172 ms. Trace:
I20250903 14:23:03.237622 1021 raft_consensus.cc:2466] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.237586 760 rpcz_store.cc:276] 0903 14:23:02.065343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.066006 (+ 663us) service_pool.cc:225] Handling call
0903 14:23:02.587505 (+521499us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:02.587863 (+ 358us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:03.223767 (+635904us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:03.237149 (+ 13382us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:03.237287 (+ 138us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:03.237413 (+ 126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:03.227576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:03.227753 (+ 177us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:03.227763 (+ 10us) write_op.cc:435] Acquired schema lock
0903 14:23:03.227767 (+ 4us) tablet.cc:660] Decoding operations
0903 14:23:03.227882 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:03.227900 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:03.227905 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:03.228016 (+ 111us) tablet.cc:699] Row locks acquired
0903 14:23:03.228018 (+ 2us) write_op.cc:260] PREPARE: finished
0903 14:23:03.228063 (+ 45us) write_op.cc:270] Start()
0903 14:23:03.228107 (+ 44us) write_op.cc:276] Timestamp: P: 1756909383227980 usec, L: 0
0903 14:23:03.228146 (+ 39us) op_driver.cc:348] REPLICATION: starting
0903 14:23:03.228385 (+ 239us) log.cc:844] Serialized 432 byte log entry
0903 14:23:03.235042 (+ 6657us) op_driver.cc:464] REPLICATION: finished
0903 14:23:03.235207 (+ 165us) write_op.cc:301] APPLY: starting
0903 14:23:03.235227 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:03.235281 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:03.235310 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:03.235486 (+ 176us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:03.235534 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:03.235545 (+ 11us) write_op.cc:312] APPLY: finished
0903 14:23:03.235641 (+ 96us) log.cc:844] Serialized 28 byte log entry
0903 14:23:03.235713 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.235740 (+ 27us) write_op.cc:454] Released schema lock
0903 14:23:03.235750 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:03.235835 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.235841 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":567433,"Hive Metastore.run_cpu_time_us":295,"Hive Metastore.run_wall_time_us":68106,"mutex_wait_us":1626,"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":606,"apply.run_wall_time_us":613,"num_ops":2,"prepare.queue_time_us":3075,"prepare.run_cpu_time_us":954,"prepare.run_wall_time_us":956,"raft.queue_time_us":269,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":256,"replication_time_us":6859}]]}
I20250903 14:23:03.237386 1413 tablet_bootstrap.cc:492] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:03.239363 1428 leader_election.cc:290] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Requested vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.240377 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6ddf1daea4a228072253412267924" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:03.240640 1298 raft_consensus.cc:3058] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.240703 1413 tablet_bootstrap.cc:654] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.243824 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6ddf1daea4a228072253412267924" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.244045 1021 raft_consensus.cc:3058] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.246908 1021 raft_consensus.cc:2466] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.243403 1298 raft_consensus.cc:2466] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.244311 1418 tablet_bootstrap.cc:492] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.250874 1413 tablet_bootstrap.cc:492] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.250890 1418 tablet_bootstrap.cc:654] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.250753 760 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_8eb8eda2d5fd408a87a960fe890af54c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:03.251017 1413 ts_tablet_manager.cc:1397] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250903 14:23:03.251796 758 catalog_manager.cc:5582] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to b03995cb49e94f108cf8caa7a502eafe (127.0.104.3). New cstate: current_term: 1 leader_uuid: "b03995cb49e94f108cf8caa7a502eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:03.248000 1095 leader_election.cc:304] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.253360 1413 raft_consensus.cc:357] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.253548 1413 raft_consensus.cc:383] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.253644 1413 raft_consensus.cc:738] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.253883 1413 consensus_queue.cc:260] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.254524 1413 ts_tablet_manager.cc:1428] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 14:23:03.257032 1428 raft_consensus.cc:2802] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.257232 1428 raft_consensus.cc:695] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:03.258064 1278 tablet_service.cc:1468] Processing CreateTablet for tablet efadfb3a6ed5403e9d17f47891ac2206 (DEFAULT_TABLE table=default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.258478 1138 tablet_service.cc:1468] Processing CreateTablet for tablet efadfb3a6ed5403e9d17f47891ac2206 (DEFAULT_TABLE table=default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.258606 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efadfb3a6ed5403e9d17f47891ac2206. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.258905 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efadfb3a6ed5403e9d17f47891ac2206. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.263845 1428 consensus_queue.cc:237] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.271842 1419 tablet_bootstrap.cc:492] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.272063 758 master_service.cc:948] ReplaceTablet: received request to replace tablet 574c4a8c94d04d4ba30965276b87a97e from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:03.273782 1419 tablet_bootstrap.cc:654] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.285737 758 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Sending DeleteTablet for 3 replicas of tablet 574c4a8c94d04d4ba30965276b87a97e
I20250903 14:23:03.287796 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet 574c4a8c94d04d4ba30965276b87a97e with delete_type TABLET_DATA_DELETED (replaced by tablet d1fcf42e749a463c8f47e275c0fe3439) from {username='slave'} at 127.0.0.1:55526
I20250903 14:23:03.288169 1420 tablet_replica.cc:331] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:03.288309 1275 tablet_service.cc:1515] Processing DeleteTablet for tablet 574c4a8c94d04d4ba30965276b87a97e with delete_type TABLET_DATA_DELETED (replaced by tablet d1fcf42e749a463c8f47e275c0fe3439) from {username='slave'} at 127.0.0.1:58704
I20250903 14:23:03.288383 1420 raft_consensus.cc:2241] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:03.288575 1421 tablet_replica.cc:331] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:03.288573 1420 raft_consensus.cc:2270] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:03.288760 1421 raft_consensus.cc:2241] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:03.289254 1421 pending_rounds.cc:70] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Trying to abort 1 pending ops.
I20250903 14:23:03.289429 1420 ts_tablet_manager.cc:1905] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:03.289462 1421 pending_rounds.cc:77] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196300833356623872 op_type: NO_OP noop_request { }
I20250903 14:23:03.289623 1421 raft_consensus.cc:2887] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 14:23:03.289772 1421 raft_consensus.cc:2270] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:03.295732 1002 tablet_service.cc:1468] Processing CreateTablet for tablet d1fcf42e749a463c8f47e275c0fe3439 (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.296238 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fcf42e749a463c8f47e275c0fe3439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.296810 1421 ts_tablet_manager.cc:1905] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:03.297026 758 catalog_manager.cc:6652] ReplaceTablet: tablet 574c4a8c94d04d4ba30965276b87a97e deleted and replaced by tablet d1fcf42e749a463c8f47e275c0fe3439
I20250903 14:23:03.298002 1139 tablet_service.cc:1515] Processing DeleteTablet for tablet 574c4a8c94d04d4ba30965276b87a97e with delete_type TABLET_DATA_DELETED (replaced by tablet d1fcf42e749a463c8f47e275c0fe3439) from {username='slave'} at 127.0.0.1:42548
I20250903 14:23:03.298346 1422 tablet_replica.cc:331] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:03.298594 1422 raft_consensus.cc:2241] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:03.298808 1422 raft_consensus.cc:2270] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:03.299674 1422 ts_tablet_manager.cc:1905] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:03.300436 1139 tablet_service.cc:1468] Processing CreateTablet for tablet d1fcf42e749a463c8f47e275c0fe3439 (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.300902 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fcf42e749a463c8f47e275c0fe3439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.301065 758 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_8eadddc09609471d8b5295f2b97e9474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:03.301735 1001 tablet_service.cc:1468] Processing CreateTablet for tablet efadfb3a6ed5403e9d17f47891ac2206 (DEFAULT_TABLE table=default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.302146 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efadfb3a6ed5403e9d17f47891ac2206. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.308706 1413 tablet_bootstrap.cc:492] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:03.310554 1413 tablet_bootstrap.cc:654] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.311297 1275 tablet_service.cc:1468] Processing CreateTablet for tablet d1fcf42e749a463c8f47e275c0fe3439 (DEFAULT_TABLE table=default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.311857 1421 ts_tablet_manager.cc:1918] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:03.313092 1422 ts_tablet_manager.cc:1918] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:03.315521 1420 ts_tablet_manager.cc:1918] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:03.315907 1422 log.cc:1199] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/574c4a8c94d04d4ba30965276b87a97e
I20250903 14:23:03.316504 1422 ts_tablet_manager.cc:1939] T 574c4a8c94d04d4ba30965276b87a97e P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:03.311967 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fcf42e749a463c8f47e275c0fe3439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.317768 1421 log.cc:1199] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/574c4a8c94d04d4ba30965276b87a97e
I20250903 14:23:03.319196 748 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet 574c4a8c94d04d4ba30965276b87a97e (table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]) successfully deleted
I20250903 14:23:03.319695 1421 ts_tablet_manager.cc:1939] T 574c4a8c94d04d4ba30965276b87a97e P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:03.320582 1420 log.cc:1199] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/574c4a8c94d04d4ba30965276b87a97e
I20250903 14:23:03.322155 1420 ts_tablet_manager.cc:1939] T 574c4a8c94d04d4ba30965276b87a97e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:03.322110 748 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet 574c4a8c94d04d4ba30965276b87a97e (table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]) successfully deleted
I20250903 14:23:03.330716 748 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet 574c4a8c94d04d4ba30965276b87a97e (table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]) successfully deleted
I20250903 14:23:03.334348 1419 tablet_bootstrap.cc:492] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.334578 1419 ts_tablet_manager.cc:1397] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.013s
I20250903 14:23:03.335557 1419 raft_consensus.cc:357] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.335814 1419 raft_consensus.cc:383] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.335912 1419 raft_consensus.cc:738] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.336232 1419 consensus_queue.cc:260] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.336817 1418 tablet_bootstrap.cc:492] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.337036 1418 ts_tablet_manager.cc:1397] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.005s
I20250903 14:23:03.337917 1419 ts_tablet_manager.cc:1428] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:03.337960 1418 raft_consensus.cc:357] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.338268 1419 tablet_bootstrap.cc:492] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.338299 1418 raft_consensus.cc:383] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.338479 1418 raft_consensus.cc:738] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.338843 1418 consensus_queue.cc:260] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.339464 1418 ts_tablet_manager.cc:1428] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 14:23:03.339825 1418 tablet_bootstrap.cc:492] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.341840 1418 tablet_bootstrap.cc:654] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.342680 1419 tablet_bootstrap.cc:654] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.345953 1419 tablet_bootstrap.cc:492] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.346200 1419 ts_tablet_manager.cc:1397] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250903 14:23:03.347307 1419 raft_consensus.cc:357] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.347632 1419 raft_consensus.cc:383] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.347738 1419 raft_consensus.cc:738] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.348127 1419 consensus_queue.cc:260] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
W20250903 14:23:03.349985 1095 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:39849, user_credentials={real_user=slave}} blocked reactor thread for 101965us
I20250903 14:23:03.350569 1413 tablet_bootstrap.cc:492] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.350795 1413 ts_tablet_manager.cc:1397] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.010s
I20250903 14:23:03.351681 1413 raft_consensus.cc:357] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.351877 1413 raft_consensus.cc:383] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.351948 1413 raft_consensus.cc:738] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.352994 1413 consensus_queue.cc:260] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.353411 1419 ts_tablet_manager.cc:1428] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250903 14:23:03.357359 1419 tablet_bootstrap.cc:492] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.358188 1413 ts_tablet_manager.cc:1428] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250903 14:23:03.359570 1419 tablet_bootstrap.cc:654] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.362915 1413 tablet_bootstrap.cc:492] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:03.365516 1413 tablet_bootstrap.cc:654] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.368257 1419 tablet_bootstrap.cc:492] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.368476 1419 ts_tablet_manager.cc:1397] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20250903 14:23:03.369405 1419 raft_consensus.cc:357] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.369638 1419 raft_consensus.cc:383] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.369735 1419 raft_consensus.cc:738] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.370050 1419 consensus_queue.cc:260] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.371922 761 catalog_manager.cc:2149] created HMS catalog entry for table [id=c8c7b6572d314c76ad67c3cab54e9b2b]
I20250903 14:23:03.373415 1419 ts_tablet_manager.cc:1428] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:23:03.378545 1418 tablet_bootstrap.cc:492] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.379374 1418 ts_tablet_manager.cc:1397] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.003s
I20250903 14:23:03.380082 761 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 1) took 1315 ms. Trace:
I20250903 14:23:03.380447 1418 raft_consensus.cc:357] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.380662 1418 raft_consensus.cc:383] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.380733 1418 raft_consensus.cc:738] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.380216 761 rpcz_store.cc:276] 0903 14:23:02.065128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.065202 (+ 74us) service_pool.cc:225] Handling call
0903 14:23:02.588098 (+522896us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:02.588560 (+ 462us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:03.371865 (+783305us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:03.379782 (+ 7917us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:03.379921 (+ 139us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:03.380035 (+ 114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:03.372685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:03.372825 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:03.372832 (+ 7us) write_op.cc:435] Acquired schema lock
0903 14:23:03.372835 (+ 3us) tablet.cc:660] Decoding operations
0903 14:23:03.372935 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:03.372989 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:03.372995 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:03.373080 (+ 85us) tablet.cc:699] Row locks acquired
0903 14:23:03.373083 (+ 3us) write_op.cc:260] PREPARE: finished
0903 14:23:03.373127 (+ 44us) write_op.cc:270] Start()
0903 14:23:03.373166 (+ 39us) write_op.cc:276] Timestamp: P: 1756909383373029 usec, L: 0
0903 14:23:03.373170 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 14:23:03.373374 (+ 204us) log.cc:844] Serialized 432 byte log entry
0903 14:23:03.378741 (+ 5367us) op_driver.cc:464] REPLICATION: finished
0903 14:23:03.379012 (+ 271us) write_op.cc:301] APPLY: starting
0903 14:23:03.379032 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:03.379083 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:03.379112 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:03.379225 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:03.379263 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:03.379271 (+ 8us) write_op.cc:312] APPLY: finished
0903 14:23:03.379363 (+ 92us) log.cc:844] Serialized 28 byte log entry
0903 14:23:03.379469 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.379493 (+ 24us) write_op.cc:454] Released schema lock
0903 14:23:03.379501 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:03.379584 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:03.379588 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":634917,"Hive Metastore.run_cpu_time_us":261,"Hive Metastore.run_wall_time_us":148100,"spinlock_wait_cycles":33792,"child_traces":[["op",{"apply.queue_time_us":215,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":561,"num_ops":2,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":827,"raft.queue_time_us":187,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":293,"replication_time_us":5534}]]}
I20250903 14:23:03.380991 1418 consensus_queue.cc:260] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.381873 761 catalog_manager.cc:5582] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2). New cstate: current_term: 1 leader_uuid: "5d4ed06db54c4b5199e190b837bc53fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:03.382737 1418 ts_tablet_manager.cc:1428] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:03.383095 1418 tablet_bootstrap.cc:492] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.385036 1413 tablet_bootstrap.cc:492] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.385214 1413 ts_tablet_manager.cc:1397] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20250903 14:23:03.385326 1418 tablet_bootstrap.cc:654] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.387882 1413 raft_consensus.cc:357] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.388180 1413 raft_consensus.cc:383] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.388288 1413 raft_consensus.cc:738] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.388576 1413 consensus_queue.cc:260] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.390838 1413 ts_tablet_manager.cc:1428] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250903 14:23:03.405983 1138 tablet_service.cc:1468] Processing CreateTablet for tablet 5a4cd5dd0f3243be9bba8defd097e2fe (DEFAULT_TABLE table=default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.406601 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a4cd5dd0f3243be9bba8defd097e2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.408476 1002 tablet_service.cc:1468] Processing CreateTablet for tablet 5a4cd5dd0f3243be9bba8defd097e2fe (DEFAULT_TABLE table=default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.408970 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a4cd5dd0f3243be9bba8defd097e2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.411878 1275 tablet_service.cc:1468] Processing CreateTablet for tablet 5a4cd5dd0f3243be9bba8defd097e2fe (DEFAULT_TABLE table=default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.412353 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a4cd5dd0f3243be9bba8defd097e2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.421833 1419 tablet_bootstrap.cc:492] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.424612 1419 tablet_bootstrap.cc:654] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.432802 1413 tablet_bootstrap.cc:492] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:03.434823 1413 tablet_bootstrap.cc:654] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.437327 1418 tablet_bootstrap.cc:492] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.437568 1418 ts_tablet_manager.cc:1397] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.055s user 0.019s sys 0.018s
I20250903 14:23:03.438078 761 master_service.cc:948] ReplaceTablet: received request to replace tablet 37f6ddf1daea4a228072253412267924 from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:03.438793 1418 raft_consensus.cc:357] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.439087 1418 raft_consensus.cc:383] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.439227 1418 raft_consensus.cc:738] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.439630 1418 consensus_queue.cc:260] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.443877 1418 ts_tablet_manager.cc:1428] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250903 14:23:03.445660 1418 tablet_bootstrap.cc:492] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.448128 1419 tablet_bootstrap.cc:492] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.448390 1419 ts_tablet_manager.cc:1397] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.007s
I20250903 14:23:03.449412 1419 raft_consensus.cc:357] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.449709 1419 raft_consensus.cc:383] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.449818 1419 raft_consensus.cc:738] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.451356 1419 consensus_queue.cc:260] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.453049 1418 tablet_bootstrap.cc:654] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.454653 761 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Sending DeleteTablet for 3 replicas of tablet 37f6ddf1daea4a228072253412267924
I20250903 14:23:03.455258 761 catalog_manager.cc:6652] ReplaceTablet: tablet 37f6ddf1daea4a228072253412267924 deleted and replaced by tablet e6522648980345ed89a6657858b83d9a
I20250903 14:23:03.455627 1419 ts_tablet_manager.cc:1428] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250903 14:23:03.459762 1459 raft_consensus.cc:491] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.459954 1459 raft_consensus.cc:513] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.460755 1459 leader_election.cc:290] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.461603 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b09ba744dc744fc3ac61383549c4babc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.461877 1021 raft_consensus.cc:2466] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.462055 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b09ba744dc744fc3ac61383549c4babc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:03.462320 1298 raft_consensus.cc:2466] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.462544 1095 leader_election.cc:304] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.462867 1459 raft_consensus.cc:2802] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.462986 1459 raft_consensus.cc:491] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.463078 1459 raft_consensus.cc:3058] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.466189 1459 raft_consensus.cc:513] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.466778 1138 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f6ddf1daea4a228072253412267924 with delete_type TABLET_DATA_DELETED (replaced by tablet e6522648980345ed89a6657858b83d9a) from {username='slave'} at 127.0.0.1:42548
I20250903 14:23:03.466763 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f6ddf1daea4a228072253412267924 with delete_type TABLET_DATA_DELETED (replaced by tablet e6522648980345ed89a6657858b83d9a) from {username='slave'} at 127.0.0.1:55526
I20250903 14:23:03.467145 1275 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f6ddf1daea4a228072253412267924 with delete_type TABLET_DATA_DELETED (replaced by tablet e6522648980345ed89a6657858b83d9a) from {username='slave'} at 127.0.0.1:58704
I20250903 14:23:03.467518 1422 tablet_replica.cc:331] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:03.467532 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b09ba744dc744fc3ac61383549c4babc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:03.467729 1422 raft_consensus.cc:2241] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:03.467751 1298 raft_consensus.cc:3058] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.468209 1422 pending_rounds.cc:70] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Trying to abort 1 pending ops.
I20250903 14:23:03.468410 1422 pending_rounds.cc:77] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196300833855090688 op_type: NO_OP noop_request { }
I20250903 14:23:03.468611 1422 raft_consensus.cc:2887] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 14:23:03.468796 1422 raft_consensus.cc:2270] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:03.469161 1420 tablet_replica.cc:331] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:03.469424 1420 raft_consensus.cc:2241] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:03.469641 1420 raft_consensus.cc:2270] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:03.470316 1420 ts_tablet_manager.cc:1905] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:03.470878 1298 raft_consensus.cc:2466] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.471556 1095 leader_election.cc:304] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.471643 1421 tablet_replica.cc:331] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:03.471871 1421 raft_consensus.cc:2241] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:03.471904 1428 raft_consensus.cc:2802] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.472077 1421 raft_consensus.cc:2270] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:03.473310 1421 ts_tablet_manager.cc:1905] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:03.475500 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b09ba744dc744fc3ac61383549c4babc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.475783 1021 raft_consensus.cc:3058] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.477078 1422 ts_tablet_manager.cc:1905] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:03.479087 1021 raft_consensus.cc:2466] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.480194 1459 leader_election.cc:290] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Requested vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.486666 1422 ts_tablet_manager.cc:1918] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:03.487200 1421 ts_tablet_manager.cc:1918] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:03.489989 1418 tablet_bootstrap.cc:492] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.491472 1422 log.cc:1199] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/37f6ddf1daea4a228072253412267924
I20250903 14:23:03.492292 1422 ts_tablet_manager.cc:1939] T 37f6ddf1daea4a228072253412267924 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:03.494465 748 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet 37f6ddf1daea4a228072253412267924 (table default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]) successfully deleted
I20250903 14:23:03.491144 1418 ts_tablet_manager.cc:1397] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.004s
I20250903 14:23:03.496095 1418 raft_consensus.cc:357] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.497128 1418 raft_consensus.cc:383] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.497468 1418 raft_consensus.cc:738] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.498011 1418 consensus_queue.cc:260] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.501291 1418 ts_tablet_manager.cc:1428] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 14:23:03.501338 1002 tablet_service.cc:1468] Processing CreateTablet for tablet e6522648980345ed89a6657858b83d9a (DEFAULT_TABLE table=default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.501866 1138 tablet_service.cc:1468] Processing CreateTablet for tablet e6522648980345ed89a6657858b83d9a (DEFAULT_TABLE table=default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.502050 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6522648980345ed89a6657858b83d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.502347 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6522648980345ed89a6657858b83d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.504392 1428 raft_consensus.cc:695] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:03.504833 1428 consensus_queue.cc:237] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.510360 1418 tablet_bootstrap.cc:492] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:03.512485 1418 tablet_bootstrap.cc:654] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.518121 1420 ts_tablet_manager.cc:1918] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:03.518337 1420 log.cc:1199] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/37f6ddf1daea4a228072253412267924
I20250903 14:23:03.518883 1420 ts_tablet_manager.cc:1939] T 37f6ddf1daea4a228072253412267924 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:03.521232 1418 tablet_bootstrap.cc:492] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:03.521509 1418 ts_tablet_manager.cc:1397] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20250903 14:23:03.522478 1418 raft_consensus.cc:357] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.522783 1418 raft_consensus.cc:383] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.522929 1418 raft_consensus.cc:738] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.523572 1418 consensus_queue.cc:260] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.524601 761 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_4aee0c2e9c0b43938b9dc11b7c6bcd9d" } modify_external_catalogs: true
I20250903 14:23:03.524910 761 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_4aee0c2e9c0b43938b9dc11b7c6bcd9d" } modify_external_catalogs: true
I20250903 14:23:03.504050 1443 raft_consensus.cc:491] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.525400 1443 raft_consensus.cc:513] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.526188 1443 leader_election.cc:290] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.526278 1418 ts_tablet_manager.cc:1428] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 14:23:03.511302 1275 tablet_service.cc:1468] Processing CreateTablet for tablet e6522648980345ed89a6657858b83d9a (DEFAULT_TABLE table=default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:03.528039 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbd068ad8e4448ba96e1bef32449a7c0" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:03.528273 1159 raft_consensus.cc:2466] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.528416 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6522648980345ed89a6657858b83d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:03.528851 1234 leader_election.cc:304] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.529453 1443 raft_consensus.cc:2802] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.529567 1443 raft_consensus.cc:491] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.529667 1443 raft_consensus.cc:3058] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.532711 1443 raft_consensus.cc:513] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.534396 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbd068ad8e4448ba96e1bef32449a7c0" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:03.534639 1159 raft_consensus.cc:3058] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.535120 1443 leader_election.cc:290] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.487418 1421 log.cc:1199] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/37f6ddf1daea4a228072253412267924
I20250903 14:23:03.537205 1421 ts_tablet_manager.cc:1939] T 37f6ddf1daea4a228072253412267924 P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:03.538407 748 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet 37f6ddf1daea4a228072253412267924 (table default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]) successfully deleted
I20250903 14:23:03.538952 1159 raft_consensus.cc:2466] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.539860 1419 tablet_bootstrap.cc:492] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:03.541327 1021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbd068ad8e4448ba96e1bef32449a7c0" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.541602 1021 raft_consensus.cc:2466] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.541960 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbd068ad8e4448ba96e1bef32449a7c0" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.542137 1022 raft_consensus.cc:3058] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.542287 1419 tablet_bootstrap.cc:654] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.543619 1234 leader_election.cc:304] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.544234 1443 raft_consensus.cc:2802] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.544466 1443 raft_consensus.cc:695] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Becoming Leader. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Running, Role: LEADER
I20250903 14:23:03.544883 1443 consensus_queue.cc:237] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.545320 1022 raft_consensus.cc:2466] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.549758 748 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet 37f6ddf1daea4a228072253412267924 (table default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]) successfully deleted
I20250903 14:23:03.556506 1413 tablet_bootstrap.cc:492] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.556753 1413 ts_tablet_manager.cc:1397] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.007s
I20250903 14:23:03.557695 1413 raft_consensus.cc:357] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.557969 1413 raft_consensus.cc:383] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.558063 1413 raft_consensus.cc:738] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.558391 1413 consensus_queue.cc:260] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.562372 1413 ts_tablet_manager.cc:1428] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250903 14:23:03.563558 1419 tablet_bootstrap.cc:492] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:03.563737 1419 ts_tablet_manager.cc:1397] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.007s
I20250903 14:23:03.564594 1419 raft_consensus.cc:357] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.564822 1413 tablet_bootstrap.cc:492] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:03.564831 1419 raft_consensus.cc:383] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.564939 1419 raft_consensus.cc:738] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.565217 1419 consensus_queue.cc:260] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.565724 1419 ts_tablet_manager.cc:1428] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:23:03.571362 1413 tablet_bootstrap.cc:654] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:03.581725 1428 raft_consensus.cc:491] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.581961 1428 raft_consensus.cc:513] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.582767 1428 leader_election.cc:290] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.583765 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5afc9676cca4a9ba1841546f5d38d7a" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.583890 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5afc9676cca4a9ba1841546f5d38d7a" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:03.584067 1022 raft_consensus.cc:2466] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.584177 1298 raft_consensus.cc:2466] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.584625 1095 leader_election.cc:304] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.588846 1428 raft_consensus.cc:2802] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.589011 1428 raft_consensus.cc:491] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.589107 1428 raft_consensus.cc:3058] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.593261 1428 raft_consensus.cc:513] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.596312 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5afc9676cca4a9ba1841546f5d38d7a" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:03.596623 1298 raft_consensus.cc:3058] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.600028 1298 raft_consensus.cc:2466] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.600822 1095 leader_election.cc:304] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.601246 1472 raft_consensus.cc:2802] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.601444 1472 raft_consensus.cc:695] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:03.601806 1472 consensus_queue.cc:237] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:03.603106 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5afc9676cca4a9ba1841546f5d38d7a" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.603358 1022 raft_consensus.cc:3058] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.606835 1022 raft_consensus.cc:2466] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.608702 1428 leader_election.cc:290] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Requested vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.609484 1413 tablet_bootstrap.cc:492] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:03.609944 1413 ts_tablet_manager.cc:1397] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.006s
I20250903 14:23:03.611181 1413 raft_consensus.cc:357] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.611868 1413 raft_consensus.cc:383] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:03.612030 1413 raft_consensus.cc:738] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:03.612404 1413 consensus_queue.cc:260] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.615257 1413 ts_tablet_manager.cc:1428] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 14:23:03.670897 1416 raft_consensus.cc:491] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.671133 1416 raft_consensus.cc:513] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.672232 1416 leader_election.cc:290] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:03.673434 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1fcf42e749a463c8f47e275c0fe3439" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:03.673491 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1fcf42e749a463c8f47e275c0fe3439" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:03.673691 1298 raft_consensus.cc:2466] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:03.673713 1159 raft_consensus.cc:2466] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:03.674273 958 leader_election.cc:304] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.674973 1416 raft_consensus.cc:2802] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.675097 1416 raft_consensus.cc:491] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.675199 1416 raft_consensus.cc:3058] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.677950 1416 raft_consensus.cc:513] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.679247 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1fcf42e749a463c8f47e275c0fe3439" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:03.679515 1159 raft_consensus.cc:3058] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.682396 1159 raft_consensus.cc:2466] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:03.684576 1416 leader_election.cc:290] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:03.685009 1416 raft_consensus.cc:491] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.685825 1416 raft_consensus.cc:513] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.685611 958 leader_election.cc:304] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.686908 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1fcf42e749a463c8f47e275c0fe3439" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:03.687129 1298 raft_consensus.cc:3058] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.690053 1298 raft_consensus.cc:2466] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:03.691766 1416 leader_election.cc:290] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:03.692559 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a4cd5dd0f3243be9bba8defd097e2fe" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:03.693281 1159 raft_consensus.cc:2466] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:03.694180 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a4cd5dd0f3243be9bba8defd097e2fe" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:03.694562 1298 raft_consensus.cc:2466] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:03.695122 1416 raft_consensus.cc:2802] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.695359 1416 raft_consensus.cc:695] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:03.695756 1416 consensus_queue.cc:237] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.693835 958 leader_election.cc:304] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.700570 1481 raft_consensus.cc:2802] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.701875 1481 raft_consensus.cc:491] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.702529 1481 raft_consensus.cc:3058] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.715790 1481 raft_consensus.cc:513] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.717203 1481 leader_election.cc:290] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:03.718981 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a4cd5dd0f3243be9bba8defd097e2fe" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:03.719223 1159 raft_consensus.cc:3058] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.722715 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a4cd5dd0f3243be9bba8defd097e2fe" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:03.723238 1298 raft_consensus.cc:3058] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.723899 1159 raft_consensus.cc:2466] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:03.724561 958 leader_election.cc:304] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.724941 1481 raft_consensus.cc:2802] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.725206 1481 raft_consensus.cc:695] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:03.725507 1481 consensus_queue.cc:237] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.727485 1298 raft_consensus.cc:2466] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:03.730887 1472 raft_consensus.cc:491] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.731181 1472 raft_consensus.cc:513] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.732347 1472 leader_election.cc:290] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:03.733855 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819c70f5de39472386cb2c27543ed8fa" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.735954 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819c70f5de39472386cb2c27543ed8fa" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:03.736553 1298 raft_consensus.cc:2391] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d4ed06db54c4b5199e190b837bc53fe in current term 1: Already voted for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in this term.
I20250903 14:23:03.737373 1095 leader_election.cc:304] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe; no voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:03.738269 1472 raft_consensus.cc:3058] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.743086 1472 raft_consensus.cc:2747] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:03.788321 1472 raft_consensus.cc:491] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.788573 1472 raft_consensus.cc:513] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.789510 1472 leader_election.cc:290] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:03.791035 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efadfb3a6ed5403e9d17f47891ac2206" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.791371 1022 raft_consensus.cc:2466] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.791859 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efadfb3a6ed5403e9d17f47891ac2206" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:03.792119 1298 raft_consensus.cc:2466] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:03.793031 1095 leader_election.cc:304] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.793747 1472 raft_consensus.cc:2802] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.793877 1472 raft_consensus.cc:491] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.793960 1472 raft_consensus.cc:3058] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.797212 1472 raft_consensus.cc:513] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.798750 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efadfb3a6ed5403e9d17f47891ac2206" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.799032 1022 raft_consensus.cc:3058] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.802306 1022 raft_consensus.cc:2466] T efadfb3a6ed5403e9d17f47891ac2206 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.803050 1095 leader_election.cc:304] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:03.803514 1428 raft_consensus.cc:2802] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.803666 1428 raft_consensus.cc:695] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:03.804030 1428 consensus_queue.cc:237] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:03.808123 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efadfb3a6ed5403e9d17f47891ac2206" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:03.808393 1298 raft_consensus.cc:3058] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.811740 1298 raft_consensus.cc:2466] T efadfb3a6ed5403e9d17f47891ac2206 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:03.812521 1472 leader_election.cc:290] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Requested vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:03.826768 1481 consensus_queue.cc:1035] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:03.852519 1443 raft_consensus.cc:491] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:03.852787 1443 raft_consensus.cc:513] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.853785 1443 leader_election.cc:290] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.855615 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0dc5829b0f9d4facb2d6da58c6be842f" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:03.855981 1158 raft_consensus.cc:2466] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.857896 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0dc5829b0f9d4facb2d6da58c6be842f" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:03.858160 1022 raft_consensus.cc:2466] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:03.858779 1234 leader_election.cc:304] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.859215 1443 raft_consensus.cc:2802] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:03.859344 1443 raft_consensus.cc:491] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:03.859460 1443 raft_consensus.cc:3058] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.860138 1416 consensus_queue.cc:1035] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:03.862447 1443 raft_consensus.cc:513] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.863155 1443 leader_election.cc:290] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:03.863936 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0dc5829b0f9d4facb2d6da58c6be842f" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:03.864148 1158 raft_consensus.cc:3058] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.865324 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0dc5829b0f9d4facb2d6da58c6be842f" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:03.865562 1022 raft_consensus.cc:3058] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:03.869302 1022 raft_consensus.cc:2466] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.871830 1234 leader_election.cc:304] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:03.873379 1158 raft_consensus.cc:2466] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:03.874245 1443 raft_consensus.cc:2802] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:03.874866 1443 raft_consensus.cc:695] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Becoming Leader. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Running, Role: LEADER
I20250903 14:23:03.875309 1443 consensus_queue.cc:237] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } }
I20250903 14:23:03.943703 756 catalog_manager.cc:3460] renamed table default.table_1bf6146990bd402ea66c580399392859 [id=c3c0e03ececf43e1a7c9c2c40acf188e] in HMS: new name default.table_31acfe71a94349f187dc3fa60a6ecc89
W20250903 14:23:03.947893 760 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8eb8eda2d5fd408a87a960fe890af54c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:03.948100 759 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_eb580089aaa843ff90baf8a5795d8f56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:03.948336 758 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8eadddc09609471d8b5295f2b97e9474 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:03.948671 753 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_867ae72d30fd4ea5b2e85874f80942ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:23:03.995393 756 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39614 (request call id 11) took 1382 ms. Trace:
I20250903 14:23:03.995654 756 rpcz_store.cc:276] 0903 14:23:02.613176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.613236 (+ 60us) service_pool.cc:225] Handling call
0903 14:23:03.429434 (+816198us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:03.994541 (+565107us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:03.994947 (+ 406us) catalog_manager.cc:3683] Apply alter schema
0903 14:23:03.995333 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":4583,"Hive Metastore.run_cpu_time_us":1443,"Hive Metastore.run_wall_time_us":509156}
I20250903 14:23:03.997515 756 catalog_manager.cc:5582] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2). New cstate: current_term: 1 leader_uuid: "5d4ed06db54c4b5199e190b837bc53fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:04.013275 756 catalog_manager.cc:5582] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to b03995cb49e94f108cf8caa7a502eafe (127.0.104.3). New cstate: current_term: 1 leader_uuid: "b03995cb49e94f108cf8caa7a502eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:04.035346 756 master_service.cc:948] ReplaceTablet: received request to replace tablet 37f6ddf1daea4a228072253412267924 from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:04.035938 756 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 28) took 1043 ms. Trace:
I20250903 14:23:04.036248 756 rpcz_store.cc:276] 0903 14:23:02.992367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:04.035269 (+1042902us) service_pool.cc:225] Handling call
0903 14:23:04.035857 (+ 588us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:04.038018 756 master_service.cc:948] ReplaceTablet: received request to replace tablet 37f6ddf1daea4a228072253412267924 from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:04.039924 756 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 29) took 1047 ms. Trace:
I20250903 14:23:04.049880 1495 tablet.cc:1722] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:04.055626 1493 tablet.cc:1722] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:04.062456 756 rpcz_store.cc:276] 0903 14:23:02.992422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:04.037910 (+1045488us) service_pool.cc:225] Handling call
0903 14:23:04.039879 (+ 1969us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:04.062897 1494 tablet.cc:1722] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:04.066800 1443 raft_consensus.cc:491] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:04.067109 1443 raft_consensus.cc:513] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.067828 1443 leader_election.cc:290] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:04.068897 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6522648980345ed89a6657858b83d9a" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:04.069141 1022 raft_consensus.cc:2466] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:04.069507 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6522648980345ed89a6657858b83d9a" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:04.069732 1159 raft_consensus.cc:2466] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:04.065740 756 catalog_manager.cc:5582] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to b03995cb49e94f108cf8caa7a502eafe (127.0.104.3). New cstate: current_term: 1 leader_uuid: "b03995cb49e94f108cf8caa7a502eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:04.070489 757 catalog_manager.cc:3460] renamed table default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13] in HMS: new name default.table_bd0820d81cd1476c9b6474c12f0053f6
I20250903 14:23:04.070843 756 catalog_manager.cc:5674] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373) does not have the latest schema for tablet 819c70f5de39472386cb2c27543ed8fa (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]). Expected version 1 got 0
I20250903 14:23:04.071295 1234 leader_election.cc:304] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:04.071918 1443 raft_consensus.cc:2802] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:04.072036 1443 raft_consensus.cc:491] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:04.072165 1443 raft_consensus.cc:3058] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:04.076728 1443 raft_consensus.cc:513] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.078213 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6522648980345ed89a6657858b83d9a" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:04.078540 1022 raft_consensus.cc:3058] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:04.083074 1022 raft_consensus.cc:2466] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:04.083828 1234 leader_election.cc:304] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:04.084303 1427 raft_consensus.cc:2802] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:04.084978 1443 leader_election.cc:290] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Requested vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:04.085518 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6522648980345ed89a6657858b83d9a" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:04.086480 1158 raft_consensus.cc:3058] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:04.087769 756 catalog_manager.cc:5582] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: HEALTHY } } }
I20250903 14:23:04.088479 756 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet 819c70f5de39472386cb2c27543ed8fa (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]). Expected version 1 got 0
I20250903 14:23:04.088774 756 catalog_manager.cc:5582] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:04.104784 1158 raft_consensus.cc:2466] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:04.110808 756 catalog_manager.cc:5674] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863) does not have the latest schema for tablet 819c70f5de39472386cb2c27543ed8fa (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]). Expected version 1 got 0
I20250903 14:23:04.113631 756 catalog_manager.cc:5582] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2). New cstate: current_term: 1 leader_uuid: "5d4ed06db54c4b5199e190b837bc53fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:04.115239 756 catalog_manager.cc:5582] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2). New cstate: current_term: 1 leader_uuid: "5d4ed06db54c4b5199e190b837bc53fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: HEALTHY } } }
I20250903 14:23:04.124125 1427 raft_consensus.cc:695] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Becoming Leader. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Running, Role: LEADER
I20250903 14:23:04.124630 1427 consensus_queue.cc:237] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.144418 756 catalog_manager.cc:6127] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e] alter complete (version 1)
W20250903 14:23:04.145469 1458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:04.146008 1458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:04.146183 1458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250903 14:23:04.148753 756 master_service.cc:948] ReplaceTablet: received request to replace tablet 819c70f5de39472386cb2c27543ed8fa from {username='slave'} at 127.0.0.1:39614
W20250903 14:23:04.160346 1458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:04.160501 1458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:04.160579 1458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:04.160638 1458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:04.163362 756 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Sending DeleteTablet for 3 replicas of tablet 819c70f5de39472386cb2c27543ed8fa
I20250903 14:23:04.165060 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet 819c70f5de39472386cb2c27543ed8fa with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22043626840419ad42d3a55eb3849) from {username='slave'} at 127.0.0.1:55526
I20250903 14:23:04.171880 1500 tablet_replica.cc:331] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:04.172245 1138 tablet_service.cc:1515] Processing DeleteTablet for tablet 819c70f5de39472386cb2c27543ed8fa with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22043626840419ad42d3a55eb3849) from {username='slave'} at 127.0.0.1:42548
I20250903 14:23:04.172281 1500 raft_consensus.cc:2241] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:04.172983 1500 raft_consensus.cc:2270] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:04.179858 1275 tablet_service.cc:1515] Processing DeleteTablet for tablet 819c70f5de39472386cb2c27543ed8fa with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22043626840419ad42d3a55eb3849) from {username='slave'} at 127.0.0.1:58704
I20250903 14:23:04.185189 1500 ts_tablet_manager.cc:1905] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:04.185742 756 catalog_manager.cc:6652] ReplaceTablet: tablet 819c70f5de39472386cb2c27543ed8fa deleted and replaced by tablet dfc22043626840419ad42d3a55eb3849
I20250903 14:23:04.185956 756 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 30) took 1193 ms. Trace:
I20250903 14:23:04.186115 756 rpcz_store.cc:276] 0903 14:23:02.992486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:04.148638 (+1156152us) service_pool.cc:225] Handling call
0903 14:23:04.185909 (+ 37271us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:04.150797 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:04.151007 (+ 210us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:04.151018 (+ 11us) write_op.cc:435] Acquired schema lock
0903 14:23:04.151021 (+ 3us) tablet.cc:660] Decoding operations
0903 14:23:04.151224 (+ 203us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:04.151248 (+ 24us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:04.151257 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:04.151399 (+ 142us) tablet.cc:699] Row locks acquired
0903 14:23:04.151404 (+ 5us) write_op.cc:260] PREPARE: finished
0903 14:23:04.151489 (+ 85us) write_op.cc:270] Start()
0903 14:23:04.151551 (+ 62us) write_op.cc:276] Timestamp: P: 1756909384151423 usec, L: 0
0903 14:23:04.151559 (+ 8us) op_driver.cc:348] REPLICATION: starting
0903 14:23:04.151872 (+ 313us) log.cc:844] Serialized 655 byte log entry
0903 14:23:04.156835 (+ 4963us) op_driver.cc:464] REPLICATION: finished
0903 14:23:04.157009 (+ 174us) write_op.cc:301] APPLY: starting
0903 14:23:04.157032 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:04.157083 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:04.157109 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:04.157243 (+ 134us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:04.157330 (+ 87us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 14:23:04.157338 (+ 8us) write_op.cc:312] APPLY: finished
0903 14:23:04.157440 (+ 102us) log.cc:844] Serialized 28 byte log entry
0903 14:23:04.157521 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:04.157548 (+ 27us) write_op.cc:454] Released schema lock
0903 14:23:04.157557 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:04.157647 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:04.157652 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":626,"apply.run_wall_time_us":626,"num_ops":2,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":1269,"prepare.run_wall_time_us":1270,"raft.queue_time_us":1443,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":401,"replication_time_us":5233}]]}
I20250903 14:23:04.193799 1502 tablet_replica.cc:331] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:04.194093 1502 raft_consensus.cc:2241] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:04.194515 1502 raft_consensus.cc:2270] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:04.195190 1501 tablet_replica.cc:331] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:04.195513 1501 raft_consensus.cc:2241] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:04.195915 1501 raft_consensus.cc:2270] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:04.197520 1501 ts_tablet_manager.cc:1905] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:04.198150 1139 tablet_service.cc:1468] Processing CreateTablet for tablet dfc22043626840419ad42d3a55eb3849 (DEFAULT_TABLE table=default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:04.198678 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc22043626840419ad42d3a55eb3849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:04.200217 1278 tablet_service.cc:1468] Processing CreateTablet for tablet dfc22043626840419ad42d3a55eb3849 (DEFAULT_TABLE table=default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:04.200695 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc22043626840419ad42d3a55eb3849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:04.205720 1427 consensus_queue.cc:1035] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.211951 1458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=41549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:04.214390 1458 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:04.216172 1500 ts_tablet_manager.cc:1918] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:23:04.216488 1500 log.cc:1199] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/819c70f5de39472386cb2c27543ed8fa
I20250903 14:23:04.217051 1500 ts_tablet_manager.cc:1939] T 819c70f5de39472386cb2c27543ed8fa P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:04.218505 748 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet 819c70f5de39472386cb2c27543ed8fa (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]) successfully deleted
I20250903 14:23:04.219763 1504 tablet_bootstrap.cc:492] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:04.220304 1002 tablet_service.cc:1468] Processing CreateTablet for tablet dfc22043626840419ad42d3a55eb3849 (DEFAULT_TABLE table=default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:04.221951 1458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 14:23:04.224802 1504 tablet_bootstrap.cc:654] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:04.229333 1427 consensus_queue.cc:1035] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.236238 1501 ts_tablet_manager.cc:1918] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:23:04.236742 1501 log.cc:1199] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/819c70f5de39472386cb2c27543ed8fa
I20250903 14:23:04.237479 1501 ts_tablet_manager.cc:1939] T 819c70f5de39472386cb2c27543ed8fa P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:04.239267 748 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet 819c70f5de39472386cb2c27543ed8fa (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]) successfully deleted
I20250903 14:23:04.239352 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc22043626840419ad42d3a55eb3849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:04.241950 1504 tablet_bootstrap.cc:492] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:04.242230 756 catalog_manager.cc:5582] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to b03995cb49e94f108cf8caa7a502eafe (127.0.104.3). New cstate: current_term: 1 leader_uuid: "b03995cb49e94f108cf8caa7a502eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: HEALTHY } } }
I20250903 14:23:04.244385 1504 ts_tablet_manager.cc:1397] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.006s
I20250903 14:23:04.245713 1504 raft_consensus.cc:357] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.245947 1504 raft_consensus.cc:383] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:04.246044 1504 raft_consensus.cc:738] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:04.246279 1504 consensus_queue.cc:260] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.250427 1504 ts_tablet_manager.cc:1428] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250903 14:23:04.251884 1502 ts_tablet_manager.cc:1905] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:04.260807 1509 tablet_bootstrap.cc:492] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:04.263464 1509 tablet_bootstrap.cc:654] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:04.266444 1502 ts_tablet_manager.cc:1918] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:23:04.267518 1502 log.cc:1199] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/819c70f5de39472386cb2c27543ed8fa
I20250903 14:23:04.268649 1502 ts_tablet_manager.cc:1939] T 819c70f5de39472386cb2c27543ed8fa P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:04.273394 756 catalog_manager.cc:5674] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373) does not have the latest schema for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]). Expected version 1 got 0
I20250903 14:23:04.273900 748 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet 819c70f5de39472386cb2c27543ed8fa (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]) successfully deleted
I20250903 14:23:04.274174 1511 tablet_bootstrap.cc:492] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
W20250903 14:23:04.275070 756 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet dfc22043626840419ad42d3a55eb3849 consensus state has no leader
W20250903 14:23:04.276062 1510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:04.278405 1509 tablet_bootstrap.cc:492] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:04.278586 1509 ts_tablet_manager.cc:1397] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20250903 14:23:04.279629 1511 tablet_bootstrap.cc:654] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:04.279497 1509 raft_consensus.cc:357] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.279724 1509 raft_consensus.cc:383] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:04.279814 1509 raft_consensus.cc:738] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:04.280141 1509 consensus_queue.cc:260] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.281826 1509 ts_tablet_manager.cc:1428] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 14:23:04.282383 756 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_293e76fad1c145eb963ab9e87c20ce43" } modify_external_catalogs: true
I20250903 14:23:04.282951 756 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_293e76fad1c145eb963ab9e87c20ce43" } modify_external_catalogs: true
I20250903 14:23:04.284199 755 catalog_manager.cc:3460] renamed table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483] in HMS: new name default.table_3368c82b8e6b4b149e8ad8d925c3d0e0
I20250903 14:23:04.286572 1511 tablet_bootstrap.cc:492] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:04.286885 1511 ts_tablet_manager.cc:1397] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20250903 14:23:04.288329 1511 raft_consensus.cc:357] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
W20250903 14:23:04.289085 1513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:04.289686 1511 raft_consensus.cc:383] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:04.289904 1511 raft_consensus.cc:738] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:04.290383 1511 consensus_queue.cc:260] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.295665 1511 ts_tablet_manager.cc:1428] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
W20250903 14:23:04.304102 1516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:04.304864 1458 server_base.cc:1047] running on GCE node
I20250903 14:23:04.306034 1458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:04.308111 1458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:04.309377 1458 hybrid_clock.cc:648] HybridClock initialized: now 1756909384309307 us; error 68 us; skew 500 ppm
I20250903 14:23:04.309897 1458 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:04.313279 1458 webserver.cc:480] Webserver started at http://127.0.104.60:41549/ using document root <none> and password file <none>
I20250903 14:23:04.314193 1458 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:04.314340 1458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:04.316538 1416 raft_consensus.cc:491] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:04.316756 1416 raft_consensus.cc:513] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.317597 1416 leader_election.cc:290] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:04.318338 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc22043626840419ad42d3a55eb3849" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:04.318588 1158 raft_consensus.cc:2466] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:04.318553 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc22043626840419ad42d3a55eb3849" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:04.318861 1298 raft_consensus.cc:2466] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:04.319597 958 leader_election.cc:304] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:04.320205 1416 raft_consensus.cc:2802] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:04.320369 1416 raft_consensus.cc:491] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:04.320477 1416 raft_consensus.cc:3058] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:04.322664 1416 raft_consensus.cc:513] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.323210 1458 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20250903 14:23:04.324156 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc22043626840419ad42d3a55eb3849" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:04.324393 1158 raft_consensus.cc:3058] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:04.327045 1158 raft_consensus.cc:2466] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:04.327641 958 leader_election.cc:304] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:04.328070 1482 raft_consensus.cc:2802] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:04.328097 1416 leader_election.cc:290] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:04.328249 1482 raft_consensus.cc:695] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:04.328869 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc22043626840419ad42d3a55eb3849" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:04.329167 1298 raft_consensus.cc:3058] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:04.329212 1482 consensus_queue.cc:237] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:04.332231 1298 raft_consensus.cc:2466] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:04.339663 1523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250903 14:23:04.340924 746 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dfc22043626840419ad42d3a55eb3849 consensus state has no leader
I20250903 14:23:04.342762 1458 fs_manager.cc:730] Time spent opening block manager: real 0.017s user 0.005s sys 0.001s
I20250903 14:23:04.343036 1458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
uuid: "2bbe08ab62bf4073825bf1b598ad2edb"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:23:04.344316 1458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:04.376925 1458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:04.378013 1458 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:04.378391 1458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:04.380051 752 catalog_manager.cc:3460] renamed table default.table_7760fb29e26f4cf88b69689632e4536b [id=69ed0433e40249699066ac4a0b3825a1] in HMS: new name default.table_2e6fe8bc133e4eb68510d15bd758709a
I20250903 14:23:04.423357 1458 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:43143
I20250903 14:23:04.423331 1574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:43143 every 8 connection(s)
I20250903 14:23:04.426160 1458 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-2/data/info.pb
I20250903 14:23:04.433238 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 1458
I20250903 14:23:04.437530 1575 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:04.444116 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Bootstrap starting.
I20250903 14:23:04.446426 1428 consensus_queue.cc:1035] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.459822 1575 log.cc:826] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:04.470382 1496 consensus_queue.cc:1035] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.505883 754 catalog_manager.cc:3460] renamed table default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328] in HMS: new name default.table_d7a178c8bc1546dd9464cf497775d66a
I20250903 14:23:04.533480 1443 consensus_queue.cc:1035] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.566633 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:23:04.567515 1482 consensus_queue.cc:1035] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.568249 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Bootstrap complete.
I20250903 14:23:04.578045 1588 consensus_queue.cc:1035] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.585219 1416 consensus_queue.cc:1035] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.592008 1588 consensus_queue.cc:1035] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.588685 1575 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:04.593719 1575 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe08ab62bf4073825bf1b598ad2edb, State: Initialized, Role: FOLLOWER
I20250903 14:23:04.594408 1205 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:04.596457 1575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:04.604779 1594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:04.605185 1594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:04.607654 1575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:23:04.610769 1068 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:04.615778 1586 consensus_queue.cc:1035] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.635596 1344 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:04.649616 1416 consensus_queue.cc:1035] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.672672 1482 consensus_queue.cc:1035] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.695894 1205 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:04.707027 1594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:04.707296 1594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:04.748919 1619 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:04.749045 1619 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: loading cluster ID for follower catalog manager: success
I20250903 14:23:04.752801 1619 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: acquiring CA information for follower catalog manager: success
I20250903 14:23:04.756474 1619 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:04.758337 1623 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:04.758673 1623 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:04.759610 1623 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:04.760090 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:04.775740 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" is_pre_election: true
I20250903 14:23:04.776676 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:04.777005 1623 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 14:23:04.790728 1427 consensus_queue.cc:1035] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.796921 1540 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:33861; Asking this server to re-register.
I20250903 14:23:04.797317 1344 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43143
I20250903 14:23:04.798195 1068 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:04.800384 1443 consensus_queue.cc:1035] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.802263 1540 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:04.802417 1539 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:49637; Asking this server to re-register.
I20250903 14:23:04.804612 1344 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:04.807282 1539 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:04.810885 1539 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:33401; Asking this server to re-register.
I20250903 14:23:04.812265 1205 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:04.814352 1539 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:04.897938 1428 consensus_queue.cc:1035] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.909348 1585 consensus_queue.cc:1035] T efadfb3a6ed5403e9d17f47891ac2206 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:04.991632 1404 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:04.991923 1404 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:05.043937 760 catalog_manager.cc:2149] created HMS catalog entry for table [id=821eb02a05ea4da5a364c5e62037f88b]
I20250903 14:23:05.091391 760 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 23) took 2178 ms. Trace:
I20250903 14:23:05.091581 760 rpcz_store.cc:276] 0903 14:23:02.912582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:03.250533 (+337951us) service_pool.cc:225] Handling call
0903 14:23:03.948826 (+698293us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:03.949420 (+ 594us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:05.043894 (+1094474us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:05.090936 (+ 47042us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:05.091114 (+ 178us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:05.091295 (+ 181us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:05.045373 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:05.045506 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:05.045513 (+ 7us) write_op.cc:435] Acquired schema lock
0903 14:23:05.045514 (+ 1us) tablet.cc:660] Decoding operations
0903 14:23:05.045610 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:05.045623 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:05.045626 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:05.045712 (+ 86us) tablet.cc:699] Row locks acquired
0903 14:23:05.045713 (+ 1us) write_op.cc:260] PREPARE: finished
0903 14:23:05.045750 (+ 37us) write_op.cc:270] Start()
0903 14:23:05.045781 (+ 31us) write_op.cc:276] Timestamp: P: 1756909385045703 usec, L: 0
0903 14:23:05.045784 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 14:23:05.045992 (+ 208us) log.cc:844] Serialized 432 byte log entry
0903 14:23:05.053352 (+ 7360us) op_driver.cc:464] REPLICATION: finished
0903 14:23:05.072764 (+ 19412us) write_op.cc:301] APPLY: starting
0903 14:23:05.072792 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:05.072859 (+ 67us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:05.072910 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:05.074480 (+ 1570us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:05.074532 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:05.074541 (+ 9us) write_op.cc:312] APPLY: finished
0903 14:23:05.075064 (+ 523us) log.cc:844] Serialized 28 byte log entry
0903 14:23:05.090722 (+ 15658us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.090753 (+ 31us) write_op.cc:454] Released schema lock
0903 14:23:05.090765 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:05.090906 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.090910 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":555999,"Hive Metastore.run_cpu_time_us":368,"Hive Metastore.run_wall_time_us":538155,"spinlock_wait_cycles":12416,"child_traces":[["op",{"apply.queue_time_us":19271,"apply.run_cpu_time_us":1002,"apply.run_wall_time_us":18134,"num_ops":2,"prepare.queue_time_us":447,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":756,"raft.queue_time_us":513,"raft.run_cpu_time_us":750,"raft.run_wall_time_us":6827,"replication_time_us":7514,"spinlock_wait_cycles":108928,"thread_start_us":19432,"threads_started":2}]]}
I20250903 14:23:05.093955 760 catalog_manager.cc:5674] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863) does not have the latest schema for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]). Expected version 1 got 0
W20250903 14:23:05.094806 760 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:44683 (request call id 20) took 810 ms (client timeout 1000 ms). Trace:
W20250903 14:23:05.094980 760 rpcz_store.cc:269] 0903 14:23:04.284458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.092632 (+808174us) service_pool.cc:225] Handling call
0903 14:23:05.094725 (+ 2093us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250903 14:23:05.106755 1139 tablet_service.cc:1468] Processing CreateTablet for tablet 9e61f697da744b0099dbd6939c1bc57c (DEFAULT_TABLE table=default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.107327 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e61f697da744b0099dbd6939c1bc57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.107456 760 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]). Expected version 1 got 0
W20250903 14:23:05.108419 760 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:42939 (request call id 23) took 813 ms (client timeout 1000 ms). Trace:
W20250903 14:23:05.108595 760 rpcz_store.cc:269] 0903 14:23:04.294557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.106485 (+811928us) service_pool.cc:225] Handling call
0903 14:23:05.108356 (+ 1871us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250903 14:23:05.112850 1002 tablet_service.cc:1468] Processing CreateTablet for tablet 9e61f697da744b0099dbd6939c1bc57c (DEFAULT_TABLE table=default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.113406 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e61f697da744b0099dbd6939c1bc57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.115921 1278 tablet_service.cc:1468] Processing CreateTablet for tablet 9e61f697da744b0099dbd6939c1bc57c (DEFAULT_TABLE table=default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.116451 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e61f697da744b0099dbd6939c1bc57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.116560 760 catalog_manager.cc:5582] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:05.116998 760 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]). Expected version 1 got 0
I20250903 14:23:05.133369 760 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_663b9a72bf8543eb90c51bd282a2db4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:05.133996 1639 tablet_bootstrap.cc:492] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:05.134229 1640 tablet_bootstrap.cc:492] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:05.138346 1639 tablet_bootstrap.cc:654] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.138336 1640 tablet_bootstrap.cc:654] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.139999 1638 tablet_bootstrap.cc:492] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:05.142069 1638 tablet_bootstrap.cc:654] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.148350 1640 tablet_bootstrap.cc:492] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:05.148569 1640 ts_tablet_manager.cc:1397] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20250903 14:23:05.149695 1640 raft_consensus.cc:357] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.149968 1640 raft_consensus.cc:383] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.150089 1640 raft_consensus.cc:738] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.150353 1640 consensus_queue.cc:260] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.151175 1638 tablet_bootstrap.cc:492] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:05.151247 1639 tablet_bootstrap.cc:492] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:05.151471 1639 ts_tablet_manager.cc:1397] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.008s
I20250903 14:23:05.151482 1638 ts_tablet_manager.cc:1397] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20250903 14:23:05.152441 1638 raft_consensus.cc:357] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.152468 1639 raft_consensus.cc:357] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.152648 1639 raft_consensus.cc:383] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.152752 1639 raft_consensus.cc:738] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.152767 1638 raft_consensus.cc:383] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.153053 1638 raft_consensus.cc:738] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.153066 1639 consensus_queue.cc:260] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.153451 1638 consensus_queue.cc:260] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.158200 1638 ts_tablet_manager.cc:1428] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 14:23:05.161756 1639 ts_tablet_manager.cc:1428] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250903 14:23:05.163137 759 catalog_manager.cc:2149] created HMS catalog entry for table [id=9c7678e144ce4862b7e1db4771a0ada2]
I20250903 14:23:05.162765 1640 ts_tablet_manager.cc:1428] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20250903 14:23:05.170086 759 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 21) took 2319 ms. Trace:
I20250903 14:23:05.170256 759 rpcz_store.cc:276] 0903 14:23:02.850349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:03.063840 (+213491us) service_pool.cc:225] Handling call
0903 14:23:03.948883 (+885043us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:03.949484 (+ 601us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:05.163070 (+1213586us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:05.169792 (+ 6722us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:05.169924 (+ 132us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:05.170040 (+ 116us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:05.164004 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:05.164156 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:05.164164 (+ 8us) write_op.cc:435] Acquired schema lock
0903 14:23:05.164169 (+ 5us) tablet.cc:660] Decoding operations
0903 14:23:05.164289 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:05.164306 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:05.164313 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:05.164399 (+ 86us) tablet.cc:699] Row locks acquired
0903 14:23:05.164401 (+ 2us) write_op.cc:260] PREPARE: finished
0903 14:23:05.164447 (+ 46us) write_op.cc:270] Start()
0903 14:23:05.164487 (+ 40us) write_op.cc:276] Timestamp: P: 1756909385164386 usec, L: 0
0903 14:23:05.164491 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 14:23:05.164705 (+ 214us) log.cc:844] Serialized 432 byte log entry
0903 14:23:05.168552 (+ 3847us) op_driver.cc:464] REPLICATION: finished
0903 14:23:05.168679 (+ 127us) write_op.cc:301] APPLY: starting
0903 14:23:05.168698 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:05.168749 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:05.168780 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:05.168903 (+ 123us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:05.168947 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:05.168955 (+ 8us) write_op.cc:312] APPLY: finished
0903 14:23:05.169057 (+ 102us) log.cc:844] Serialized 28 byte log entry
0903 14:23:05.169149 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.169175 (+ 26us) write_op.cc:454] Released schema lock
0903 14:23:05.169182 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:05.169266 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.169271 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1094153,"Hive Metastore.run_cpu_time_us":416,"Hive Metastore.run_wall_time_us":119151,"child_traces":[["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":578,"apply.run_wall_time_us":578,"num_ops":2,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":823,"raft.queue_time_us":1389,"raft.run_cpu_time_us":578,"raft.run_wall_time_us":577,"replication_time_us":4022}]]}
I20250903 14:23:05.185182 1002 tablet_service.cc:1468] Processing CreateTablet for tablet 05b219b49aa24cbe995c03d1098875dc (DEFAULT_TABLE table=default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.185683 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b219b49aa24cbe995c03d1098875dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.187050 1278 tablet_service.cc:1468] Processing CreateTablet for tablet 05b219b49aa24cbe995c03d1098875dc (DEFAULT_TABLE table=default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.187621 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b219b49aa24cbe995c03d1098875dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.192194 1639 tablet_bootstrap.cc:492] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:05.192587 1139 tablet_service.cc:1468] Processing CreateTablet for tablet 05b219b49aa24cbe995c03d1098875dc (DEFAULT_TABLE table=default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.193080 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b219b49aa24cbe995c03d1098875dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.194198 1639 tablet_bootstrap.cc:654] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.205946 1638 tablet_bootstrap.cc:492] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:05.208290 1640 tablet_bootstrap.cc:492] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:05.210248 1638 tablet_bootstrap.cc:654] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.210297 1640 tablet_bootstrap.cc:654] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.217257 1639 tablet_bootstrap.cc:492] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:05.218147 1639 ts_tablet_manager.cc:1397] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.011s
I20250903 14:23:05.219165 1639 raft_consensus.cc:357] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.220165 1639 raft_consensus.cc:383] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.220386 1639 raft_consensus.cc:738] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.220805 1639 consensus_queue.cc:260] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.221860 758 catalog_manager.cc:2149] created HMS catalog entry for table [id=23986959b59c41b190cf66a723f57e63]
I20250903 14:23:05.226886 1639 ts_tablet_manager.cc:1428] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250903 14:23:05.228140 1638 tablet_bootstrap.cc:492] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:05.228343 1638 ts_tablet_manager.cc:1397] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20250903 14:23:05.229061 758 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 25) took 2239 ms. Trace:
I20250903 14:23:05.229229 758 rpcz_store.cc:276] 0903 14:23:02.989496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:03.300819 (+311323us) service_pool.cc:225] Handling call
0903 14:23:03.949559 (+648740us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:03.949981 (+ 422us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:05.221801 (+1271820us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:05.228744 (+ 6943us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:05.228875 (+ 131us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:05.229001 (+ 126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:05.222661 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:05.222795 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:05.222802 (+ 7us) write_op.cc:435] Acquired schema lock
0903 14:23:05.222805 (+ 3us) tablet.cc:660] Decoding operations
0903 14:23:05.222936 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:05.222951 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:05.222956 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:05.223040 (+ 84us) tablet.cc:699] Row locks acquired
0903 14:23:05.223043 (+ 3us) write_op.cc:260] PREPARE: finished
0903 14:23:05.223085 (+ 42us) write_op.cc:270] Start()
0903 14:23:05.223124 (+ 39us) write_op.cc:276] Timestamp: P: 1756909385223020 usec, L: 0
0903 14:23:05.223128 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 14:23:05.223330 (+ 202us) log.cc:844] Serialized 432 byte log entry
0903 14:23:05.227859 (+ 4529us) op_driver.cc:464] REPLICATION: finished
0903 14:23:05.228041 (+ 182us) write_op.cc:301] APPLY: starting
0903 14:23:05.228062 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:05.228121 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:05.228175 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:05.228305 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:05.228345 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:05.228353 (+ 8us) write_op.cc:312] APPLY: finished
0903 14:23:05.228441 (+ 88us) log.cc:844] Serialized 28 byte log entry
0903 14:23:05.228522 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.228547 (+ 25us) write_op.cc:454] Released schema lock
0903 14:23:05.228555 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:05.228636 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.228640 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1212970,"Hive Metastore.run_cpu_time_us":325,"Hive Metastore.run_wall_time_us":58437,"child_traces":[["op",{"apply.queue_time_us":124,"apply.run_cpu_time_us":586,"apply.run_wall_time_us":586,"num_ops":2,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":819,"raft.queue_time_us":479,"raft.run_cpu_time_us":465,"raft.run_wall_time_us":463,"replication_time_us":4692,"spinlock_wait_cycles":16768}]]}
I20250903 14:23:05.231750 1638 raft_consensus.cc:357] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.232318 1638 raft_consensus.cc:383] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.232718 1638 raft_consensus.cc:738] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.233362 1638 consensus_queue.cc:260] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.249186 1638 ts_tablet_manager.cc:1428] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20250903 14:23:05.249806 1640 tablet_bootstrap.cc:492] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:05.250094 1640 ts_tablet_manager.cc:1397] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.000s
I20250903 14:23:05.252547 758 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_d5a7e44d7d1c4005aa2bc39c855d2418" } modify_external_catalogs: true
I20250903 14:23:05.252811 758 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_d5a7e44d7d1c4005aa2bc39c855d2418" } modify_external_catalogs: true
I20250903 14:23:05.252878 1002 tablet_service.cc:1468] Processing CreateTablet for tablet 8ec2abff20f1413fad17fd393d8fd1e9 (DEFAULT_TABLE table=default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.252854 1139 tablet_service.cc:1468] Processing CreateTablet for tablet 8ec2abff20f1413fad17fd393d8fd1e9 (DEFAULT_TABLE table=default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.253369 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec2abff20f1413fad17fd393d8fd1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.253480 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec2abff20f1413fad17fd393d8fd1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.256273 1278 tablet_service.cc:1468] Processing CreateTablet for tablet 8ec2abff20f1413fad17fd393d8fd1e9 (DEFAULT_TABLE table=default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.256788 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec2abff20f1413fad17fd393d8fd1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.263518 1640 raft_consensus.cc:357] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.263828 1640 raft_consensus.cc:383] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.263957 1640 raft_consensus.cc:738] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.266471 1640 consensus_queue.cc:260] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.269366 1639 tablet_bootstrap.cc:492] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:05.271368 1639 tablet_bootstrap.cc:654] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.272119 1638 tablet_bootstrap.cc:492] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:05.273855 1640 ts_tablet_manager.cc:1428] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 14:23:05.274175 1638 tablet_bootstrap.cc:654] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.280066 1640 tablet_bootstrap.cc:492] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:05.282124 1640 tablet_bootstrap.cc:654] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.288432 1638 tablet_bootstrap.cc:492] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:05.288707 1638 ts_tablet_manager.cc:1397] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20250903 14:23:05.289556 1638 raft_consensus.cc:357] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.291232 1638 raft_consensus.cc:383] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.291409 1638 raft_consensus.cc:738] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.292335 1638 consensus_queue.cc:260] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.296548 1482 consensus_queue.cc:1035] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:05.296965 1640 tablet_bootstrap.cc:492] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:05.297143 1640 ts_tablet_manager.cc:1397] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250903 14:23:05.298065 1640 raft_consensus.cc:357] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.298269 1640 raft_consensus.cc:383] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.298367 1640 raft_consensus.cc:738] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.298599 1640 consensus_queue.cc:260] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.299234 1640 ts_tablet_manager.cc:1428] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 14:23:05.302702 753 catalog_manager.cc:2149] created HMS catalog entry for table [id=548535eb537048f489876c6c19d999f5]
I20250903 14:23:05.304872 759 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_0ba72da4f2b84fb39612871a39e8d19f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:05.310927 1639 tablet_bootstrap.cc:492] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:05.312088 753 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 20) took 2496 ms. Trace:
I20250903 14:23:05.312216 753 rpcz_store.cc:276] 0903 14:23:02.816006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.933426 (+117420us) service_pool.cc:225] Handling call
0903 14:23:03.950100 (+1016674us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:03.950507 (+ 407us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:05.302649 (+1352142us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:05.311760 (+ 9111us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 14:23:05.311882 (+ 122us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 14:23:05.312002 (+ 120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:05.303563 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 14:23:05.303696 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0903 14:23:05.303704 (+ 8us) write_op.cc:435] Acquired schema lock
0903 14:23:05.303707 (+ 3us) tablet.cc:660] Decoding operations
0903 14:23:05.303821 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0903 14:23:05.303836 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0903 14:23:05.303842 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0903 14:23:05.303934 (+ 92us) tablet.cc:699] Row locks acquired
0903 14:23:05.303937 (+ 3us) write_op.cc:260] PREPARE: finished
0903 14:23:05.303981 (+ 44us) write_op.cc:270] Start()
0903 14:23:05.304023 (+ 42us) write_op.cc:276] Timestamp: P: 1756909385303948 usec, L: 0
0903 14:23:05.304029 (+ 6us) op_driver.cc:348] REPLICATION: starting
0903 14:23:05.304282 (+ 253us) log.cc:844] Serialized 432 byte log entry
0903 14:23:05.310558 (+ 6276us) op_driver.cc:464] REPLICATION: finished
0903 14:23:05.311018 (+ 460us) write_op.cc:301] APPLY: starting
0903 14:23:05.311041 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0903 14:23:05.311095 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0903 14:23:05.311133 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 14:23:05.311297 (+ 164us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 14:23:05.311341 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 14:23:05.311349 (+ 8us) write_op.cc:312] APPLY: finished
0903 14:23:05.311438 (+ 89us) log.cc:844] Serialized 28 byte log entry
0903 14:23:05.311543 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.311569 (+ 26us) write_op.cc:454] Released schema lock
0903 14:23:05.311577 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 14:23:05.311653 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:05.311657 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1270943,"Hive Metastore.run_cpu_time_us":341,"Hive Metastore.run_wall_time_us":81044,"spinlock_wait_cycles":8064,"child_traces":[["op",{"apply.queue_time_us":380,"apply.run_cpu_time_us":634,"apply.run_wall_time_us":634,"num_ops":2,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":843,"prepare.run_wall_time_us":844,"raft.queue_time_us":1330,"raft.run_cpu_time_us":559,"raft.run_wall_time_us":559,"replication_time_us":6494}]]}
I20250903 14:23:05.315117 1639 ts_tablet_manager.cc:1397] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.001s
I20250903 14:23:05.316430 1639 raft_consensus.cc:357] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.317907 1639 raft_consensus.cc:383] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.319033 1496 consensus_queue.cc:1035] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:05.325141 753 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_219a289c9b7a44af91e47feece52778a" } modify_external_catalogs: true
I20250903 14:23:05.326164 1278 tablet_service.cc:1468] Processing CreateTablet for tablet e6db03a8c88941278f0ab15599e4f975 (DEFAULT_TABLE table=default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.326260 1139 tablet_service.cc:1468] Processing CreateTablet for tablet e6db03a8c88941278f0ab15599e4f975 (DEFAULT_TABLE table=default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.326310 753 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_219a289c9b7a44af91e47feece52778a" } modify_external_catalogs: true
I20250903 14:23:05.326678 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6db03a8c88941278f0ab15599e4f975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.326678 1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6db03a8c88941278f0ab15599e4f975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.326005 1002 tablet_service.cc:1468] Processing CreateTablet for tablet e6db03a8c88941278f0ab15599e4f975 (DEFAULT_TABLE table=default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:05.336944 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6db03a8c88941278f0ab15599e4f975. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:05.351073 1640 tablet_bootstrap.cc:492] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:05.354511 1640 tablet_bootstrap.cc:654] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.356029 1639 raft_consensus.cc:738] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.356729 1639 consensus_queue.cc:260] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.358408 1638 ts_tablet_manager.cc:1428] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.070s user 0.002s sys 0.000s
I20250903 14:23:05.364261 1427 consensus_queue.cc:1035] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:05.365131 1639 ts_tablet_manager.cc:1428] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20250903 14:23:05.365523 1639 tablet_bootstrap.cc:492] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:05.366619 1640 tablet_bootstrap.cc:492] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:05.366887 1640 ts_tablet_manager.cc:1397] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20250903 14:23:05.367868 1640 raft_consensus.cc:357] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.368181 1640 raft_consensus.cc:383] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.368309 1640 raft_consensus.cc:738] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.368438 1638 tablet_bootstrap.cc:492] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:05.368712 1640 consensus_queue.cc:260] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.369760 1640 ts_tablet_manager.cc:1428] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 14:23:05.372113 1638 tablet_bootstrap.cc:654] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.372656 1639 tablet_bootstrap.cc:654] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:05.388978 1638 tablet_bootstrap.cc:492] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:05.389294 1638 ts_tablet_manager.cc:1397] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.005s
I20250903 14:23:05.390300 1638 raft_consensus.cc:357] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.390622 1638 raft_consensus.cc:383] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.390703 1639 tablet_bootstrap.cc:492] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:05.390755 1638 raft_consensus.cc:738] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.390944 1639 ts_tablet_manager.cc:1397] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.008s
I20250903 14:23:05.391186 1638 consensus_queue.cc:260] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.391846 1639 raft_consensus.cc:357] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.391996 1638 ts_tablet_manager.cc:1428] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:05.392153 1639 raft_consensus.cc:383] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:05.392273 1639 raft_consensus.cc:738] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:05.392556 1639 consensus_queue.cc:260] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.393708 1639 ts_tablet_manager.cc:1428] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:05.405190 1482 consensus_queue.cc:1035] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:05.429904 1587 raft_consensus.cc:491] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:05.430124 1587 raft_consensus.cc:513] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.430830 1587 leader_election.cc:290] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:05.431753 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b219b49aa24cbe995c03d1098875dc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:05.431754 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b219b49aa24cbe995c03d1098875dc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:05.431998 1022 raft_consensus.cc:2466] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:05.432128 1298 raft_consensus.cc:2466] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:05.432708 1095 leader_election.cc:304] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:05.433270 1587 raft_consensus.cc:2802] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:05.433385 1587 raft_consensus.cc:491] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:05.433465 1587 raft_consensus.cc:3058] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.452384 1587 raft_consensus.cc:513] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.453953 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b219b49aa24cbe995c03d1098875dc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:05.454213 1022 raft_consensus.cc:3058] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.457391 1022 raft_consensus.cc:2466] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:05.458318 1095 leader_election.cc:304] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:05.470780 1587 leader_election.cc:290] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Requested vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:05.471165 1587 raft_consensus.cc:2802] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:05.471369 1587 raft_consensus.cc:695] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:05.471979 1587 consensus_queue.cc:237] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:05.472343 1496 raft_consensus.cc:491] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:05.472523 1496 raft_consensus.cc:513] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.473239 1496 leader_election.cc:290] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:05.473611 1482 raft_consensus.cc:491] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:05.473814 1482 raft_consensus.cc:513] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.474318 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6db03a8c88941278f0ab15599e4f975" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:05.474588 1022 raft_consensus.cc:2466] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:05.474625 1482 leader_election.cc:290] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:05.475136 1234 leader_election.cc:304] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:05.475832 1496 raft_consensus.cc:2802] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:05.475893 1297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e61f697da744b0099dbd6939c1bc57c" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:05.475999 1496 raft_consensus.cc:491] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:05.476104 1496 raft_consensus.cc:3058] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.476177 1297 raft_consensus.cc:2466] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:05.479907 958 leader_election.cc:304] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:05.479856 1496 raft_consensus.cc:513] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.480304 1482 raft_consensus.cc:2802] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:05.480429 1482 raft_consensus.cc:491] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:05.480548 1482 raft_consensus.cc:3058] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.480985 1496 leader_election.cc:290] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Requested vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:05.481343 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6db03a8c88941278f0ab15599e4f975" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:05.481590 1022 raft_consensus.cc:3058] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.473887 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6db03a8c88941278f0ab15599e4f975" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:05.475533 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e61f697da744b0099dbd6939c1bc57c" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:05.483793 1159 raft_consensus.cc:2466] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 0.
I20250903 14:23:05.483796 1157 raft_consensus.cc:2466] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:05.484493 1159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6db03a8c88941278f0ab15599e4f975" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:05.484753 1159 raft_consensus.cc:3058] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.486124 1482 raft_consensus.cc:513] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.487937 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e61f697da744b0099dbd6939c1bc57c" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:05.488251 1157 raft_consensus.cc:3058] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.488755 1159 raft_consensus.cc:2466] T e6db03a8c88941278f0ab15599e4f975 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
I20250903 14:23:05.492100 1234 leader_election.cc:304] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:05.492763 1496 raft_consensus.cc:2802] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:05.493485 1496 raft_consensus.cc:695] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Becoming Leader. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Running, Role: LEADER
I20250903 14:23:05.493849 1496 consensus_queue.cc:237] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.493983 1297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e61f697da744b0099dbd6939c1bc57c" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:05.494182 1297 raft_consensus.cc:3058] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.494622 1157 raft_consensus.cc:2466] T 9e61f697da744b0099dbd6939c1bc57c P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:05.495214 958 leader_election.cc:304] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:05.495678 1663 tablet.cc:1722] T dfc22043626840419ad42d3a55eb3849 P b03995cb49e94f108cf8caa7a502eafe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:05.495798 1660 raft_consensus.cc:2802] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:05.497356 1662 tablet.cc:1722] T dfc22043626840419ad42d3a55eb3849 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:05.497581 1297 raft_consensus.cc:2466] T 9e61f697da744b0099dbd6939c1bc57c P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:05.497660 1482 leader_election.cc:290] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:05.498957 1664 tablet.cc:1722] T dfc22043626840419ad42d3a55eb3849 P 5d4ed06db54c4b5199e190b837bc53fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:05.500483 1660 raft_consensus.cc:695] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:05.500876 1660 consensus_queue.cc:237] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.502724 1022 raft_consensus.cc:2466] T e6db03a8c88941278f0ab15599e4f975 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03995cb49e94f108cf8caa7a502eafe in term 1.
W20250903 14:23:05.503382 1095 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:39849, user_credentials={real_user=slave}} blocked reactor thread for 45079.1us
I20250903 14:23:05.505259 1297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b219b49aa24cbe995c03d1098875dc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:05.505786 1297 raft_consensus.cc:3058] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.509758 1297 raft_consensus.cc:2466] T 05b219b49aa24cbe995c03d1098875dc P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:05.510470 1656 raft_consensus.cc:491] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:05.511272 1656 raft_consensus.cc:513] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.513044 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ec2abff20f1413fad17fd393d8fd1e9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:05.513358 1158 raft_consensus.cc:2466] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:05.513927 958 leader_election.cc:304] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:05.514299 1667 raft_consensus.cc:2802] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:05.514406 1667 raft_consensus.cc:491] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:05.514545 1667 raft_consensus.cc:3058] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.516000 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ec2abff20f1413fad17fd393d8fd1e9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:05.516315 1298 raft_consensus.cc:2466] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:05.516899 1656 leader_election.cc:290] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:05.530179 1667 raft_consensus.cc:513] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.531388 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ec2abff20f1413fad17fd393d8fd1e9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:05.531661 1158 raft_consensus.cc:3058] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.534540 1667 leader_election.cc:290] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:05.534920 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ec2abff20f1413fad17fd393d8fd1e9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:05.535144 1298 raft_consensus.cc:3058] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:05.537318 1298 raft_consensus.cc:2466] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:05.537817 958 leader_election.cc:304] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:05.538375 1660 raft_consensus.cc:2802] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:05.538605 1660 raft_consensus.cc:695] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:05.539368 1660 consensus_queue.cc:237] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:05.570300 1158 raft_consensus.cc:2466] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:05.824540 1673 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:05.825088 1673 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:05.831472 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:05.830952 1673 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:05.832837 1674 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:05.833045 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" is_pre_election: true
I20250903 14:23:05.833312 1674 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:05.833997 1550 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e83636ce3ca47f3bfc9f84f6755eec2 in term 1.
I20250903 14:23:05.834658 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: f4c58d929e53460883d05e0f14f20780
I20250903 14:23:05.835007 1673 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 14:23:05.835157 1673 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:05.835284 1673 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 1 FOLLOWER]: Advancing to term 2
I20250903 14:23:05.836913 1674 leader_election.cc:290] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437)
I20250903 14:23:05.837304 701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" is_pre_election: true
I20250903 14:23:05.838192 1673 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:05.838647 701 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2bbe08ab62bf4073825bf1b598ad2edb in current term 2: Already voted for candidate 2e83636ce3ca47f3bfc9f84f6755eec2 in this term.
I20250903 14:23:05.839008 1673 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 election: Requested vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:05.840624 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780"
I20250903 14:23:05.841166 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:05.844480 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "2bbe08ab62bf4073825bf1b598ad2edb"
I20250903 14:23:05.844789 1550 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 1 FOLLOWER]: Advancing to term 2
I20250903 14:23:05.850039 1550 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e83636ce3ca47f3bfc9f84f6755eec2 in term 2.
I20250903 14:23:05.850694 1527 leader_election.cc:304] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb; no voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:05.851117 1674 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 14:23:05.851625 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: f4c58d929e53460883d05e0f14f20780
I20250903 14:23:05.852377 1673 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 14:23:05.853793 1673 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 LEADER]: Becoming Leader. State: Replica: 2e83636ce3ca47f3bfc9f84f6755eec2, State: Running, Role: LEADER
I20250903 14:23:05.854310 1673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:05.856251 701 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 LEADER]: Rejecting Update request from peer f4c58d929e53460883d05e0f14f20780 for earlier term 1. Current term is 2. Ops: []
I20250903 14:23:05.856755 1676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e83636ce3ca47f3bfc9f84f6755eec2. Latest consensus state: current_term: 2 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:05.857237 1676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:05.857470 1675 consensus_queue.cc:1046] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250903 14:23:05.862434 1675 consensus_queue.cc:1230] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Peer 2e83636ce3ca47f3bfc9f84f6755eec2 log is divergent from this leader: its last log entry 2.55 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 54
I20250903 14:23:05.862941 1678 raft_consensus.cc:3053] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 14:23:05.863144 1678 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Running, Role: LEADER
I20250903 14:23:05.863713 1678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:05.864287 1678 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 1 FOLLOWER]: Advancing to term 2
I20250903 14:23:05.871229 1677 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:23:05.874752 1677 catalog_manager.cc:671] Loaded metadata for table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]
I20250903 14:23:05.875505 1677 catalog_manager.cc:671] Loaded metadata for table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483]
I20250903 14:23:05.876040 1677 catalog_manager.cc:671] Loaded metadata for table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]
I20250903 14:23:05.876533 1677 catalog_manager.cc:671] Loaded metadata for table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]
I20250903 14:23:05.877290 1677 catalog_manager.cc:671] Loaded metadata for table default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]
I20250903 14:23:05.877895 1677 catalog_manager.cc:671] Loaded metadata for table default.table_7760fb29e26f4cf88b69689632e4536b [id=69ed0433e40249699066ac4a0b3825a1]
I20250903 14:23:05.878583 1677 catalog_manager.cc:671] Loaded metadata for table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]
I20250903 14:23:05.879153 1677 catalog_manager.cc:671] Loaded metadata for table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]
I20250903 14:23:05.879734 1677 catalog_manager.cc:671] Loaded metadata for table default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13]
I20250903 14:23:05.880300 1677 catalog_manager.cc:671] Loaded metadata for table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]
I20250903 14:23:05.880780 1677 catalog_manager.cc:671] Loaded metadata for table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]
I20250903 14:23:05.881258 1677 catalog_manager.cc:671] Loaded metadata for table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]
I20250903 14:23:05.881771 1677 catalog_manager.cc:671] Loaded metadata for table default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328]
I20250903 14:23:05.882314 1677 catalog_manager.cc:671] Loaded metadata for table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]
I20250903 14:23:05.884990 1677 tablet_loader.cc:96] loaded metadata for tablet 05b219b49aa24cbe995c03d1098875dc (table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2])
I20250903 14:23:05.885612 1677 tablet_loader.cc:96] loaded metadata for tablet 0dc5829b0f9d4facb2d6da58c6be842f (table default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d])
I20250903 14:23:05.890427 1677 tablet_loader.cc:96] loaded metadata for tablet 5a4cd5dd0f3243be9bba8defd097e2fe (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b])
I20250903 14:23:05.891037 1677 tablet_loader.cc:96] loaded metadata for tablet 8ec2abff20f1413fad17fd393d8fd1e9 (table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63])
I20250903 14:23:05.891569 1677 tablet_loader.cc:96] loaded metadata for tablet 9e61f697da744b0099dbd6939c1bc57c (table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b])
I20250903 14:23:05.891947 1677 tablet_loader.cc:96] loaded metadata for tablet b09ba744dc744fc3ac61383549c4babc (table default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328])
I20250903 14:23:05.892241 1677 tablet_loader.cc:96] loaded metadata for tablet b5afc9676cca4a9ba1841546f5d38d7a (table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9])
I20250903 14:23:05.892560 1677 tablet_loader.cc:96] loaded metadata for tablet c51713a4e7554285ba265825b8651efe (table default.table_7760fb29e26f4cf88b69689632e4536b [id=69ed0433e40249699066ac4a0b3825a1])
I20250903 14:23:05.893061 1677 tablet_loader.cc:96] loaded metadata for tablet cbd068ad8e4448ba96e1bef32449a7c0 (table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417])
I20250903 14:23:05.893456 1677 tablet_loader.cc:96] loaded metadata for tablet d1fcf42e749a463c8f47e275c0fe3439 (table default.table_466c344e6f85402282f897ca4ddb8406 [id=2224cdb559fe45599b2464369cd2b483])
I20250903 14:23:05.894086 1677 tablet_loader.cc:96] loaded metadata for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e])
I20250903 14:23:05.894594 1677 tablet_loader.cc:96] loaded metadata for tablet e6522648980345ed89a6657858b83d9a (table default.table_6b6dbda5e4a448e6b56eb740d7d99a15 [id=8590620c66374843adbe167d46c1de13])
I20250903 14:23:05.895016 1677 tablet_loader.cc:96] loaded metadata for tablet e6db03a8c88941278f0ab15599e4f975 (table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5])
I20250903 14:23:05.895368 1677 tablet_loader.cc:96] loaded metadata for tablet efadfb3a6ed5403e9d17f47891ac2206 (table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d])
I20250903 14:23:05.895654 1677 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:23:05.896786 1677 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:05.896883 1677 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:23:05.898809 1677 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:23:05.900166 1677 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Loaded TSK: 0
I20250903 14:23:05.901542 1677 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:23:05.903172 1677 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 14:23:05.904846 1677 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 11
I20250903 14:23:05.940239 761 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]
W20250903 14:23:05.948431 924 catalog_manager.cc:3897] Illegal state: an error occurred while updating the sys-catalog: replica f4c58d929e53460883d05e0f14f20780 is not leader of this config: current role FOLLOWER
W20250903 14:23:05.948746 924 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 12 ALTER_TABLE default.table_bd0820d81cd1476c9b6474c12f0053f6: Illegal state: an error occurred while updating the sys-catalog: replica f4c58d929e53460883d05e0f14f20780 is not leader of this config: current role FOLLOWER
W20250903 14:23:05.948925 924 hms_notification_log_listener.cc:143] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica f4c58d929e53460883d05e0f14f20780 is not leader of this config: current role FOLLOWER
I20250903 14:23:05.949246 757 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39614 (request call id 19) took 3159 ms. Trace:
I20250903 14:23:05.949399 757 rpcz_store.cc:276] 0903 14:23:02.789533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.794185 (+ 4652us) service_pool.cc:225] Handling call
0903 14:23:03.947591 (+1153406us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.949191 (+2001600us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":5808,"Hive Metastore.run_cpu_time_us":889,"Hive Metastore.run_wall_time_us":116595}
I20250903 14:23:05.949683 761 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39614 (request call id 27) took 2957 ms. Trace:
I20250903 14:23:05.949791 761 rpcz_store.cc:276] 0903 14:23:02.992299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:03.524236 (+531937us) service_pool.cc:225] Handling call
0903 14:23:03.994165 (+469929us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.949595 (+1955430us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1308261,"Hive Metastore.run_cpu_time_us":502,"Hive Metastore.run_wall_time_us":637463}
I20250903 14:23:05.949985 755 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39614 (request call id 12) took 3288 ms. Trace:
I20250903 14:23:05.950140 755 rpcz_store.cc:276] 0903 14:23:02.661311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.661417 (+ 106us) service_pool.cc:225] Handling call
0903 14:23:03.947332 (+1285915us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.949367 (+2002035us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.949632 (+ 265us) catalog_manager.cc:3683] Apply alter schema
0903 14:23:05.949936 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":122614,"Hive Metastore.run_cpu_time_us":962,"Hive Metastore.run_wall_time_us":213931,"mutex_wait_us":158}
I20250903 14:23:05.950686 754 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39614 (request call id 22) took 3070 ms. Trace:
I20250903 14:23:05.950841 754 rpcz_store.cc:276] 0903 14:23:02.879923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:03.185456 (+305533us) service_pool.cc:225] Handling call
0903 14:23:03.947336 (+761880us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.950174 (+2002838us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.950415 (+ 241us) catalog_manager.cc:3683] Apply alter schema
0903 14:23:05.950638 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":431316,"Hive Metastore.run_cpu_time_us":845,"Hive Metastore.run_wall_time_us":125548}
I20250903 14:23:05.951511 754 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 38) took 2561 ms. Trace:
I20250903 14:23:05.951551 755 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 39) took 2501 ms. Trace:
I20250903 14:23:05.951699 755 rpcz_store.cc:276] 0903 14:23:03.450159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.951245 (+2501086us) service_pool.cc:225] Handling call
0903 14:23:05.951500 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.951946 755 master_service.cc:948] ReplaceTablet: received request to replace tablet 819c70f5de39472386cb2c27543ed8fa from {username='slave'} at 127.0.0.1:39614
W20250903 14:23:05.952105 760 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_663b9a72bf8543eb90c51bd282a2db4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:23:05.952327 755 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 40) took 2450 ms. Trace:
I20250903 14:23:05.952427 755 rpcz_store.cc:276] 0903 14:23:03.501877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.951876 (+2449999us) service_pool.cc:225] Handling call
0903 14:23:05.952253 (+ 377us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250903 14:23:05.952838 1354 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:40437)
I20250903 14:23:05.952947 755 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 41) took 2364 ms. Trace:
I20250903 14:23:05.953066 755 rpcz_store.cc:276] 0903 14:23:03.588084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.952624 (+2364540us) service_pool.cc:225] Handling call
0903 14:23:05.952883 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.953290 755 master_service.cc:948] ReplaceTablet: received request to replace tablet b09ba744dc744fc3ac61383549c4babc from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:05.953578 755 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 42) took 2285 ms. Trace:
I20250903 14:23:05.953675 755 rpcz_store.cc:276] 0903 14:23:03.667659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.953209 (+2285550us) service_pool.cc:225] Handling call
0903 14:23:05.953548 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250903 14:23:05.954457 759 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0ba72da4f2b84fb39612871a39e8d19f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:23:05.960039 691 catalog_manager.cc:5582] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:05.961398 691 catalog_manager.cc:5582] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:05.951515 757 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 36) took 2671 ms. Trace:
I20250903 14:23:05.962294 757 rpcz_store.cc:276] 0903 14:23:03.280129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.951095 (+2670966us) service_pool.cc:225] Handling call
0903 14:23:05.951461 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.951648 754 rpcz_store.cc:276] 0903 14:23:03.390044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.951150 (+2561106us) service_pool.cc:225] Handling call
0903 14:23:05.951457 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57600}
I20250903 14:23:05.962129 752 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39614 (request call id 18) took 3242 ms. Trace:
I20250903 14:23:05.964124 752 rpcz_store.cc:276] 0903 14:23:02.707406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:02.785960 (+ 78554us) service_pool.cc:225] Handling call
0903 14:23:03.947329 (+1161369us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.949540 (+2002211us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:05.949794 (+ 254us) catalog_manager.cc:3683] Apply alter schema
0903 14:23:05.950078 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":335784,"Hive Metastore.run_cpu_time_us":795,"Hive Metastore.run_wall_time_us":95672}
I20250903 14:23:05.964519 752 master_service.cc:948] ReplaceTablet: received request to replace tablet 0dc5829b0f9d4facb2d6da58c6be842f from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:05.964586 754 master_service.cc:948] ReplaceTablet: received request to replace tablet e6522648980345ed89a6657858b83d9a from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:05.964866 752 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 44) took 1721 ms. Trace:
I20250903 14:23:05.965000 752 rpcz_store.cc:276] 0903 14:23:04.243386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.964433 (+1721047us) service_pool.cc:225] Handling call
0903 14:23:05.964814 (+ 381us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.965204 752 master_service.cc:948] ReplaceTablet: received request to replace tablet 5a4cd5dd0f3243be9bba8defd097e2fe from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:05.965492 752 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 46) took 1568 ms. Trace:
I20250903 14:23:05.951577 761 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39614 (request call id 37) took 2563 ms. Trace:
I20250903 14:23:05.965585 752 rpcz_store.cc:276] 0903 14:23:04.397038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.965153 (+1568115us) service_pool.cc:225] Handling call
0903 14:23:05.965444 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.966161 690 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46974:
table { table_name: "default.Table_6b6dbda5e4a448e6b56eb740d7d99a15" } new_table_name: "default.Table_bd0820d81cd1476c9b6474c12f0053f6" modify_external_catalogs: true
I20250903 14:23:05.965646 761 rpcz_store.cc:276] 0903 14:23:03.387699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.951150 (+2563451us) service_pool.cc:225] Handling call
0903 14:23:05.951514 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28032}
I20250903 14:23:05.967325 752 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:39628 (request call id 0) took 1194 ms. Trace:
I20250903 14:23:05.973666 1550 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Refusing update from remote peer 2e83636ce3ca47f3bfc9f84f6755eec2: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 2 index: 56. (index mismatch)
I20250903 14:23:05.974570 761 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:45734 (request call id 5) took 1048 ms. Trace:
I20250903 14:23:05.974707 761 rpcz_store.cc:276] 0903 14:23:04.919044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.967204 (+1048160us) service_pool.cc:225] Handling call
0903 14:23:05.967504 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.974682 1676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
W20250903 14:23:05.975047 1354 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:37899)
W20250903 14:23:05.975493 1354 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:37899)
I20250903 14:23:05.970172 752 rpcz_store.cc:276] 0903 14:23:04.773173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.967023 (+1193850us) service_pool.cc:225] Handling call
0903 14:23:05.967267 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.964877 754 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39614 (request call id 45) took 1695 ms. Trace:
I20250903 14:23:05.978520 754 rpcz_store.cc:276] 0903 14:23:04.269022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.964534 (+1695512us) service_pool.cc:225] Handling call
0903 14:23:05.964814 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52480}
W20250903 14:23:05.978868 1354 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:37899)
I20250903 14:23:05.979045 770 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 2 FOLLOWER]: Refusing update from remote peer 2e83636ce3ca47f3bfc9f84f6755eec2: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 2 index: 56. (index mismatch)
I20250903 14:23:05.980613 688 master_service.cc:948] ReplaceTablet: received request to replace tablet 819c70f5de39472386cb2c27543ed8fa from {username='slave'} at 127.0.0.1:46974
I20250903 14:23:05.981597 1676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250903 14:23:05.981961 685 master_service.cc:948] ReplaceTablet: received request to replace tablet b09ba744dc744fc3ac61383549c4babc from {username='slave'} at 127.0.0.1:46974
I20250903 14:23:05.982054 686 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_152f6fe6ff314070b2b909c120bf6a10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:05.983029 684 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_a8c65bc030834131a6a9b071e7cae292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:05.983974 683 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_6fd4c0ea3f684a06b60c606e942d3961"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:05.984917 755 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39614 (request call id 43) took 1784 ms. Trace:
I20250903 14:23:05.985067 755 rpcz_store.cc:276] 0903 14:23:04.200140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.967830 (+1767690us) service_pool.cc:225] Handling call
0903 14:23:05.984866 (+ 17036us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 14:23:05.986896 689 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_d63d2c9adbc5421f9a1c01f637efe55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:05.988070 1458 master_runner.cc:232] Existing masters have differing terms: 1 vs 2
I20250903 14:23:05.988360 1458 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:05.993464 1674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e83636ce3ca47f3bfc9f84f6755eec2. Latest consensus state: current_term: 2 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:05.993783 1674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:05.999351 687 master_service.cc:948] ReplaceTablet: received request to replace tablet 0dc5829b0f9d4facb2d6da58c6be842f from {username='slave'} at 127.0.0.1:46974
I20250903 14:23:06.001699 688 master_service.cc:948] ReplaceTablet: received request to replace tablet 5a4cd5dd0f3243be9bba8defd097e2fe from {username='slave'} at 127.0.0.1:46974
I20250903 14:23:06.003141 1678 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e83636ce3ca47f3bfc9f84f6755eec2. Latest consensus state: current_term: 2 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:06.003468 1678 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:06.018469 1678 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:06.018734 1678 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:06.019493 682 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_0b07d90eb033424fab95da7de937a939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:06.022724 1673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:06.022996 1673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:06.023313 729 master_runner.cc:232] Existing masters have differing terms: 1 vs 2
I20250903 14:23:06.023535 729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:06.024423 1673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:06.024629 1673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:06.037302 660 master_runner.cc:263] No leader master found from master 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:06.037456 660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:06.048583 685 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Sending DeleteTablet for 3 replicas of tablet b09ba744dc744fc3ac61383549c4babc
I20250903 14:23:06.049935 1674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:06.050163 1674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:06.052304 691 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_b56d836b85da4bd4bb8b347606cd1334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:06.052688 687 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Sending DeleteTablet for 3 replicas of tablet 0dc5829b0f9d4facb2d6da58c6be842f
I20250903 14:23:06.053102 687 catalog_manager.cc:6652] ReplaceTablet: tablet 0dc5829b0f9d4facb2d6da58c6be842f deleted and replaced by tablet 888cbb3e8e724c5f8b19bf129868cea9
I20250903 14:23:06.055877 687 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_dc1dfd28029e4b3498cd0f42e3227d17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:06.056398 685 catalog_manager.cc:6652] ReplaceTablet: tablet b09ba744dc744fc3ac61383549c4babc deleted and replaced by tablet f2f7427e774b42b1bc3de1e70792f7c0
I20250903 14:23:06.057119 685 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46974:
table { table_name: "default.Table_6955211ac77d4f9cb6e5314ebdebf3ce" } modify_external_catalogs: true
I20250903 14:23:06.057418 685 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46974:
table { table_name: "default.Table_6955211ac77d4f9cb6e5314ebdebf3ce" } modify_external_catalogs: true
I20250903 14:23:06.073696 688 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Sending DeleteTablet for 3 replicas of tablet 5a4cd5dd0f3243be9bba8defd097e2fe
I20250903 14:23:06.074223 688 catalog_manager.cc:6652] ReplaceTablet: tablet 5a4cd5dd0f3243be9bba8defd097e2fe deleted and replaced by tablet 9d67a811c1734c13b9119ed59e8ad1fb
I20250903 14:23:06.074707 688 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46974:
name: "default.Table_1b9ce2c993734dcfa81dc61644f332bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:23:06.104871 906 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2f7427e774b42b1bc3de1e70792f7c0 consensus state has no leader
I20250903 14:23:06.105199 756 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]
I20250903 14:23:06.105769 756 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39614 (request call id 31) took 3023 ms. Trace:
I20250903 14:23:06.105911 756 rpcz_store.cc:276] 0903 14:23:03.082055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:04.282210 (+1200155us) service_pool.cc:225] Handling call
0903 14:23:05.949303 (+1667093us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:06.105728 (+156425us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":12719,"Hive Metastore.run_cpu_time_us":194,"Hive Metastore.run_wall_time_us":142086}
I20250903 14:23:06.107165 906 catalog_manager.cc:2747] Deleting table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417] in response to Hive Metastore notification log event 20
I20250903 14:23:06.116878 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet b09ba744dc744fc3ac61383549c4babc with delete_type TABLET_DATA_DELETED (replaced by tablet f2f7427e774b42b1bc3de1e70792f7c0) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.117630 906 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Sending DeleteTablet for 3 replicas of tablet cbd068ad8e4448ba96e1bef32449a7c0
I20250903 14:23:06.120025 1001 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f with delete_type TABLET_DATA_DELETED (replaced by tablet 888cbb3e8e724c5f8b19bf129868cea9) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.120327 1000 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a4cd5dd0f3243be9bba8defd097e2fe with delete_type TABLET_DATA_DELETED (replaced by tablet 9d67a811c1734c13b9119ed59e8ad1fb) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.120985 1000 tablet_service.cc:1468] Processing CreateTablet for tablet f2f7427e774b42b1bc3de1e70792f7c0 (DEFAULT_TABLE table=default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.121286 999 tablet_service.cc:1468] Processing CreateTablet for tablet 888cbb3e8e724c5f8b19bf129868cea9 (DEFAULT_TABLE table=default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.121529 1000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f7427e774b42b1bc3de1e70792f7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.121791 997 tablet_service.cc:1515] Processing DeleteTablet for tablet cbd068ad8e4448ba96e1bef32449a7c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:23:06 UTC) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.122018 999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 888cbb3e8e724c5f8b19bf129868cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.123943 998 tablet_service.cc:1468] Processing CreateTablet for tablet 9d67a811c1734c13b9119ed59e8ad1fb (DEFAULT_TABLE table=default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.124466 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d67a811c1734c13b9119ed59e8ad1fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.130789 1278 tablet_service.cc:1515] Processing DeleteTablet for tablet b09ba744dc744fc3ac61383549c4babc with delete_type TABLET_DATA_DELETED (replaced by tablet f2f7427e774b42b1bc3de1e70792f7c0) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.131251 1275 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f with delete_type TABLET_DATA_DELETED (replaced by tablet 888cbb3e8e724c5f8b19bf129868cea9) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.131532 1276 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f with delete_type TABLET_DATA_DELETED (replaced by tablet 888cbb3e8e724c5f8b19bf129868cea9) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.131865 1273 tablet_service.cc:1468] Processing CreateTablet for tablet f2f7427e774b42b1bc3de1e70792f7c0 (DEFAULT_TABLE table=default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.132035 1271 tablet_service.cc:1515] Processing DeleteTablet for tablet cbd068ad8e4448ba96e1bef32449a7c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:23:06 UTC) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.132429 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f7427e774b42b1bc3de1e70792f7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.132951 1274 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a4cd5dd0f3243be9bba8defd097e2fe with delete_type TABLET_DATA_DELETED (replaced by tablet 9d67a811c1734c13b9119ed59e8ad1fb) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.134377 1276 tablet_service.cc:1468] Processing CreateTablet for tablet 9d67a811c1734c13b9119ed59e8ad1fb (DEFAULT_TABLE table=default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.134907 1276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d67a811c1734c13b9119ed59e8ad1fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.136119 1275 tablet_service.cc:1468] Processing CreateTablet for tablet 888cbb3e8e724c5f8b19bf129868cea9 (DEFAULT_TABLE table=default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.136533 1275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 888cbb3e8e724c5f8b19bf129868cea9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 14:23:06.152240 675 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2f7427e774b42b1bc3de1e70792f7c0 consensus state has no leader
I20250903 14:23:06.153309 690 master_service.cc:948] ReplaceTablet: received request to replace tablet e6522648980345ed89a6657858b83d9a from {username='slave'} at 127.0.0.1:46974
I20250903 14:23:06.156308 1698 tablet_replica.cc:331] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:06.156637 1698 raft_consensus.cc:2241] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.156939 1698 raft_consensus.cc:2270] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.157750 1698 ts_tablet_manager.cc:1905] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.160022 1700 tablet_bootstrap.cc:492] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:06.160398 906 catalog_manager.cc:2747] Deleting table default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d] in response to Hive Metastore notification log event 21
I20250903 14:23:06.162106 1700 tablet_bootstrap.cc:654] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.170651 690 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Sending DeleteTablet for 3 replicas of tablet e6522648980345ed89a6657858b83d9a
I20250903 14:23:06.171165 690 catalog_manager.cc:6652] ReplaceTablet: tablet e6522648980345ed89a6657858b83d9a deleted and replaced by tablet 3a86369c9b3b4321abe742146a37623b
I20250903 14:23:06.171583 690 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46974:
table { table_name: "default.Table_31acfe71a94349f187dc3fa60a6ecc89" } new_table_name: "default.Table_2c0dd73e42f04addad73b6089d755a89" modify_external_catalogs: true
I20250903 14:23:06.174525 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet e6522648980345ed89a6657858b83d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a86369c9b3b4321abe742146a37623b) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.176298 906 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Sending DeleteTablet for 3 replicas of tablet 888cbb3e8e724c5f8b19bf129868cea9
W20250903 14:23:06.177733 686 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_152f6fe6ff314070b2b909c120bf6a10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:06.179584 684 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a8c65bc030834131a6a9b071e7cae292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:06.180714 683 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6fd4c0ea3f684a06b60c606e942d3961 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:06.181794 689 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d63d2c9adbc5421f9a1c01f637efe55b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:23:06.181869 1698 ts_tablet_manager.cc:1918] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.181988 1698 log.cc:1199] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/b09ba744dc744fc3ac61383549c4babc
I20250903 14:23:06.182435 1698 ts_tablet_manager.cc:1939] T b09ba744dc744fc3ac61383549c4babc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
W20250903 14:23:06.182825 682 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0b07d90eb033424fab95da7de937a939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:06.183835 691 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b56d836b85da4bd4bb8b347606cd1334 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:06.184878 687 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dc1dfd28029e4b3498cd0f42e3227d17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 14:23:06.186196 688 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1b9ce2c993734dcfa81dc61644f332bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 14:23:06.188058 1276 tablet_service.cc:1515] Processing DeleteTablet for tablet e6522648980345ed89a6657858b83d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a86369c9b3b4321abe742146a37623b) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.194648 678 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet b09ba744dc744fc3ac61383549c4babc (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]) successfully deleted
I20250903 14:23:06.195289 1273 tablet_service.cc:1515] Processing DeleteTablet for tablet 888cbb3e8e724c5f8b19bf129868cea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:23:06 UTC) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.195757 999 tablet_service.cc:1515] Processing DeleteTablet for tablet 888cbb3e8e724c5f8b19bf129868cea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:23:06 UTC) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.201007 1139 tablet_service.cc:1515] Processing DeleteTablet for tablet b09ba744dc744fc3ac61383549c4babc with delete_type TABLET_DATA_DELETED (replaced by tablet f2f7427e774b42b1bc3de1e70792f7c0) from {username='slave'} at 127.0.0.1:42558
I20250903 14:23:06.201674 1138 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f with delete_type TABLET_DATA_DELETED (replaced by tablet 888cbb3e8e724c5f8b19bf129868cea9) from {username='slave'} at 127.0.0.1:42558
I20250903 14:23:06.201604 1134 tablet_service.cc:1468] Processing CreateTablet for tablet 9d67a811c1734c13b9119ed59e8ad1fb (DEFAULT_TABLE table=default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.201758 1135 tablet_service.cc:1468] Processing CreateTablet for tablet 888cbb3e8e724c5f8b19bf129868cea9 (DEFAULT_TABLE table=default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.201953 1137 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a4cd5dd0f3243be9bba8defd097e2fe with delete_type TABLET_DATA_DELETED (replaced by tablet 9d67a811c1734c13b9119ed59e8ad1fb) from {username='slave'} at 127.0.0.1:42558
I20250903 14:23:06.202203 1134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d67a811c1734c13b9119ed59e8ad1fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.202410 1133 tablet_service.cc:1515] Processing DeleteTablet for tablet cbd068ad8e4448ba96e1bef32449a7c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:23:06 UTC) from {username='slave'} at 127.0.0.1:42558
I20250903 14:23:06.202486 1136 tablet_service.cc:1468] Processing CreateTablet for tablet f2f7427e774b42b1bc3de1e70792f7c0 (DEFAULT_TABLE table=default.table_ace62b8a67ba4aac89cb28cf186a6816 [id=d237316ba9fd4366a139c31dd9524328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.202739 1131 tablet_service.cc:1515] Processing DeleteTablet for tablet 888cbb3e8e724c5f8b19bf129868cea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 14:23:06 UTC) from {username='slave'} at 127.0.0.1:42558
I20250903 14:23:06.202972 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f7427e774b42b1bc3de1e70792f7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.203486 760 catalog_manager.cc:2149] created HMS catalog entry for table [id=3379b0e151684210ac6845928dea7e14]
W20250903 14:23:06.204334 760 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica f4c58d929e53460883d05e0f14f20780 is not leader of this config: current role FOLLOWER
I20250903 14:23:06.205806 1132 tablet_service.cc:1515] Processing DeleteTablet for tablet e6522648980345ed89a6657858b83d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a86369c9b3b4321abe742146a37623b) from {username='slave'} at 127.0.0.1:42558
I20250903 14:23:06.206545 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 888cbb3e8e724c5f8b19bf129868cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.213884 1273 tablet_service.cc:1468] Processing CreateTablet for tablet 3a86369c9b3b4321abe742146a37623b (DEFAULT_TABLE table=default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.214304 1132 tablet_service.cc:1468] Processing CreateTablet for tablet 3a86369c9b3b4321abe742146a37623b (DEFAULT_TABLE table=default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.214495 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a86369c9b3b4321abe742146a37623b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.214866 1132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a86369c9b3b4321abe742146a37623b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.223984 1001 tablet_service.cc:1468] Processing CreateTablet for tablet 3a86369c9b3b4321abe742146a37623b (DEFAULT_TABLE table=default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.224629 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a86369c9b3b4321abe742146a37623b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.231339 1700 tablet_bootstrap.cc:492] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:06.231662 1700 ts_tablet_manager.cc:1397] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.000s
I20250903 14:23:06.232643 1700 raft_consensus.cc:357] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.233036 1700 raft_consensus.cc:383] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.233145 1700 raft_consensus.cc:738] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.233994 1700 consensus_queue.cc:260] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.234823 1343 heartbeater.cc:499] Master 127.0.104.62:37899 was elected leader, sending a full tablet report...
I20250903 14:23:06.238266 1706 tablet_bootstrap.cc:492] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:06.241293 1067 heartbeater.cc:499] Master 127.0.104.62:37899 was elected leader, sending a full tablet report...
I20250903 14:23:06.241550 1700 ts_tablet_manager.cc:1428] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250903 14:23:06.236814 1698 tablet_replica.cc:331] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:06.242393 1700 tablet_bootstrap.cc:492] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:06.243592 1698 raft_consensus.cc:2241] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.243831 1698 raft_consensus.cc:2270] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.244877 1700 tablet_bootstrap.cc:654] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.255669 1707 tablet_replica.cc:331] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:06.256023 1707 raft_consensus.cc:2241] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:06.260955 1698 ts_tablet_manager.cc:1905] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.268985 1706 tablet_bootstrap.cc:654] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.272207 1707 raft_consensus.cc:2270] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.277940 1707 ts_tablet_manager.cc:1905] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.261358 1708 tablet_bootstrap.cc:492] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:06.278630 1715 tablet.cc:1722] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.293558 1708 tablet_bootstrap.cc:654] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.271574 1701 tablet_replica.cc:331] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:06.296470 1701 raft_consensus.cc:2241] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.297260 1701 raft_consensus.cc:2270] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.302531 1701 ts_tablet_manager.cc:1905] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.305318 1719 tablet.cc:1722] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.328497 1724 tablet.cc:1722] T c51713a4e7554285ba265825b8651efe P b03995cb49e94f108cf8caa7a502eafe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.329087 926 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:06.330216 1723 tablet.cc:1722] T d1fcf42e749a463c8f47e275c0fe3439 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.330767 1715 tablet.cc:1722] T d1fcf42e749a463c8f47e275c0fe3439 P 5d4ed06db54c4b5199e190b837bc53fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.333618 1725 tablet.cc:1722] T c51713a4e7554285ba265825b8651efe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.338073 1700 tablet_bootstrap.cc:492] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:06.338368 1700 ts_tablet_manager.cc:1397] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.096s user 0.019s sys 0.039s
I20250903 14:23:06.339516 1700 raft_consensus.cc:357] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.339855 1700 raft_consensus.cc:383] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.340018 1700 raft_consensus.cc:738] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.340423 1700 consensus_queue.cc:260] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.341454 1700 ts_tablet_manager.cc:1428] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 14:23:06.357095 1698 ts_tablet_manager.cc:1918] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.357419 1698 log.cc:1199] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/0dc5829b0f9d4facb2d6da58c6be842f
I20250903 14:23:06.358085 1698 ts_tablet_manager.cc:1939] T 0dc5829b0f9d4facb2d6da58c6be842f P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:06.359143 1706 tablet_bootstrap.cc:492] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:06.359402 1706 ts_tablet_manager.cc:1397] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.121s user 0.008s sys 0.000s
I20250903 14:23:06.360406 1706 raft_consensus.cc:357] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.360702 1706 raft_consensus.cc:383] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.360863 1706 raft_consensus.cc:738] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.361292 1706 consensus_queue.cc:260] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.367391 1729 tablet.cc:1722] T d1fcf42e749a463c8f47e275c0fe3439 P b03995cb49e94f108cf8caa7a502eafe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.372619 1708 tablet_bootstrap.cc:492] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:06.372907 1708 ts_tablet_manager.cc:1397] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.002s
I20250903 14:23:06.373875 1708 raft_consensus.cc:357] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.377386 1708 raft_consensus.cc:383] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.379285 1708 raft_consensus.cc:738] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.380951 1708 consensus_queue.cc:260] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.382916 1706 ts_tablet_manager.cc:1428] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250903 14:23:06.383528 1708 ts_tablet_manager.cc:1428] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250903 14:23:06.359136 1700 tablet_bootstrap.cc:492] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:06.393589 1700 tablet_bootstrap.cc:654] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.396814 1724 tablet.cc:1722] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.397819 1706 tablet_bootstrap.cc:492] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:06.398983 1708 tablet_bootstrap.cc:492] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:06.393519 1715 tablet.cc:1722] T c51713a4e7554285ba265825b8651efe P 5d4ed06db54c4b5199e190b837bc53fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 14:23:06.382468 1707 ts_tablet_manager.cc:1918] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.401139 1706 tablet_bootstrap.cc:654] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.401459 1707 log.cc:1199] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/b09ba744dc744fc3ac61383549c4babc
I20250903 14:23:06.402025 1707 ts_tablet_manager.cc:1939] T b09ba744dc744fc3ac61383549c4babc P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:06.405542 1698 tablet_replica.cc:331] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:06.406773 1707 tablet_replica.cc:331] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:06.406939 678 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet b09ba744dc744fc3ac61383549c4babc (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]) successfully deleted
I20250903 14:23:06.376600 1701 ts_tablet_manager.cc:1918] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.407195 1700 tablet_bootstrap.cc:492] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:06.407280 1701 log.cc:1199] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/b09ba744dc744fc3ac61383549c4babc
I20250903 14:23:06.407843 1701 ts_tablet_manager.cc:1939] T b09ba744dc744fc3ac61383549c4babc P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:06.407224 1698 raft_consensus.cc:2241] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:06.411231 1698 raft_consensus.cc:2270] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.412433 1698 ts_tablet_manager.cc:1905] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.411801 1708 tablet_bootstrap.cc:654] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.412770 1707 raft_consensus.cc:2241] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.414258 1707 raft_consensus.cc:2270] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.414355 678 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet b09ba744dc744fc3ac61383549c4babc (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]) successfully deleted
I20250903 14:23:06.414925 1701 tablet_replica.cc:331] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:06.415194 1701 raft_consensus.cc:2241] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:06.415685 1701 raft_consensus.cc:2270] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.415922 1707 ts_tablet_manager.cc:1905] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.416332 1701 ts_tablet_manager.cc:1905] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.416805 1700 ts_tablet_manager.cc:1397] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.011s
I20250903 14:23:06.417882 1700 raft_consensus.cc:357] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.418143 1700 raft_consensus.cc:383] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.418238 1700 raft_consensus.cc:738] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.418534 1700 consensus_queue.cc:260] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.424129 1706 tablet_bootstrap.cc:492] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:06.424350 1706 ts_tablet_manager.cc:1397] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.007s
I20250903 14:23:06.425251 1706 raft_consensus.cc:357] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.425472 1706 raft_consensus.cc:383] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.425596 1706 raft_consensus.cc:738] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.425882 1706 consensus_queue.cc:260] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.426632 1706 ts_tablet_manager.cc:1428] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250903 14:23:06.427263 1700 ts_tablet_manager.cc:1428] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250903 14:23:06.428387 1708 tablet_bootstrap.cc:492] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:06.428575 1708 ts_tablet_manager.cc:1397] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250903 14:23:06.429872 1708 raft_consensus.cc:357] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.430104 1708 raft_consensus.cc:383] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.430188 1708 raft_consensus.cc:738] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.430305 686 catalog_manager.cc:2149] created HMS catalog entry for table [id=4ad47717c2424f57a3235322430e8d3c]
I20250903 14:23:06.430459 1708 consensus_queue.cc:260] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.442816 1706 tablet_bootstrap.cc:492] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:06.445134 1706 tablet_bootstrap.cc:654] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.449225 1709 consensus_queue.cc:1035] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.462091 686 catalog_manager.cc:5674] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373) does not have the latest schema for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]). Expected version 1 got 0
I20250903 14:23:06.464531 1701 ts_tablet_manager.cc:1918] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.464757 1701 log.cc:1199] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/0dc5829b0f9d4facb2d6da58c6be842f
I20250903 14:23:06.465317 1701 ts_tablet_manager.cc:1939] T 0dc5829b0f9d4facb2d6da58c6be842f P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:06.466784 686 catalog_manager.cc:5582] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to b03995cb49e94f108cf8caa7a502eafe (127.0.104.3). New cstate: current_term: 1 leader_uuid: "b03995cb49e94f108cf8caa7a502eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: HEALTHY } } }
I20250903 14:23:06.469348 686 catalog_manager.cc:5674] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373) does not have the latest schema for tablet c51713a4e7554285ba265825b8651efe (table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1]). Expected version 1 got 0
I20250903 14:23:06.469730 686 catalog_manager.cc:5674] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373) does not have the latest schema for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]). Expected version 1 got 0
I20250903 14:23:06.469451 1708 ts_tablet_manager.cc:1428] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20250903 14:23:06.470417 686 catalog_manager.cc:5674] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373) does not have the latest schema for tablet d1fcf42e749a463c8f47e275c0fe3439 (table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483]). Expected version 1 got 0
I20250903 14:23:06.476696 1700 tablet_bootstrap.cc:492] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:06.477133 1701 tablet_replica.cc:331] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:06.477632 1701 raft_consensus.cc:2241] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:06.478366 1701 raft_consensus.cc:2270] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.479578 1701 ts_tablet_manager.cc:1905] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.479833 753 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]
I20250903 14:23:06.480437 753 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39614 (request call id 35) took 3228 ms. Trace:
I20250903 14:23:06.481765 753 rpcz_store.cc:276] 0903 14:23:03.251755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.324924 (+2073169us) service_pool.cc:225] Handling call
0903 14:23:05.949351 (+624427us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:06.480391 (+531040us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":253786,"Hive Metastore.run_cpu_time_us":380,"Hive Metastore.run_wall_time_us":257638,"mutex_wait_us":3594}
I20250903 14:23:06.486565 1671 consensus_queue.cc:1035] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.500624 1136 tablet_service.cc:1468] Processing CreateTablet for tablet 0e78a7c3701f464080b7b887574a6b7a (DEFAULT_TABLE table=default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.500964 1001 tablet_service.cc:1468] Processing CreateTablet for tablet 0e78a7c3701f464080b7b887574a6b7a (DEFAULT_TABLE table=default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.501190 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e78a7c3701f464080b7b887574a6b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.501443 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e78a7c3701f464080b7b887574a6b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.503870 1273 tablet_service.cc:1468] Processing CreateTablet for tablet 0e78a7c3701f464080b7b887574a6b7a (DEFAULT_TABLE table=default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.504401 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e78a7c3701f464080b7b887574a6b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.509974 1700 tablet_bootstrap.cc:654] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.515667 1278 tablet_service.cc:1515] Processing DeleteTablet for tablet b09ba744dc744fc3ac61383549c4babc with delete_type TABLET_DATA_DELETED (replaced by tablet f2f7427e774b42b1bc3de1e70792f7c0) from {username='slave'} at 127.0.0.1:58708
I20250903 14:23:06.516479 1278 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f with delete_type TABLET_DATA_DELETED (replaced by tablet 888cbb3e8e724c5f8b19bf129868cea9) from {username='slave'} at 127.0.0.1:58708
W20250903 14:23:06.516767 686 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3a86369c9b3b4321abe742146a37623b consensus state has no leader
I20250903 14:23:06.520640 1708 tablet_bootstrap.cc:492] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:06.529357 1698 ts_tablet_manager.cc:1918] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.531250 686 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]). Expected version 1 got 0
I20250903 14:23:06.532516 1700 tablet_bootstrap.cc:492] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:06.533182 1700 ts_tablet_manager.cc:1397] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.002s
I20250903 14:23:06.534130 1706 tablet_bootstrap.cc:492] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:06.534202 1700 raft_consensus.cc:357] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.534510 1700 raft_consensus.cc:383] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.534605 1700 raft_consensus.cc:738] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.534556 1706 ts_tablet_manager.cc:1397] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.005s
I20250903 14:23:06.534871 1700 consensus_queue.cc:260] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.535560 1700 ts_tablet_manager.cc:1428] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 14:23:06.535614 1710 raft_consensus.cc:491] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:06.535778 1710 raft_consensus.cc:513] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.535970 1700 tablet_bootstrap.cc:492] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:06.535919 1706 raft_consensus.cc:357] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.536113 1706 raft_consensus.cc:383] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.536222 1706 raft_consensus.cc:738] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.536492 1706 consensus_queue.cc:260] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.536778 1710 leader_election.cc:290] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.537459 686 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet c51713a4e7554285ba265825b8651efe (table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1]). Expected version 1 got 0
I20250903 14:23:06.537817 1706 ts_tablet_manager.cc:1428] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250903 14:23:06.537806 686 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]). Expected version 1 got 0
I20250903 14:23:06.538162 686 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet d1fcf42e749a463c8f47e275c0fe3439 (table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483]). Expected version 1 got 0
I20250903 14:23:06.538147 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "888cbb3e8e724c5f8b19bf129868cea9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:06.538357 1298 raft_consensus.cc:2466] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:06.539085 958 leader_election.cc:304] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:06.539534 1710 raft_consensus.cc:2802] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:06.539820 1710 raft_consensus.cc:491] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:06.539898 1710 raft_consensus.cc:3058] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.532956 1698 log.cc:1199] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/5a4cd5dd0f3243be9bba8defd097e2fe
I20250903 14:23:06.541649 1698 ts_tablet_manager.cc:1939] T 5a4cd5dd0f3243be9bba8defd097e2fe P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:06.542855 1698 tablet_replica.cc:331] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:06.543098 1698 raft_consensus.cc:2241] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.543347 678 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet 5a4cd5dd0f3243be9bba8defd097e2fe (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]) successfully deleted
I20250903 14:23:06.544186 1700 tablet_bootstrap.cc:654] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.544373 1706 tablet_bootstrap.cc:492] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:06.545709 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "888cbb3e8e724c5f8b19bf129868cea9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
W20250903 14:23:06.547607 958 leader_election.cc:343] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Illegal state: must be running to vote when last-logged opid is not known
I20250903 14:23:06.547861 1001 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dc5829b0f9d4facb2d6da58c6be842f with delete_type TABLET_DATA_DELETED (replaced by tablet 888cbb3e8e724c5f8b19bf129868cea9) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.543290 1698 raft_consensus.cc:2270] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.548856 1698 ts_tablet_manager.cc:1905] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.549409 1710 raft_consensus.cc:513] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.552138 1701 ts_tablet_manager.cc:1918] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.552626 1001 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a4cd5dd0f3243be9bba8defd097e2fe with delete_type TABLET_DATA_DELETED (replaced by tablet 9d67a811c1734c13b9119ed59e8ad1fb) from {username='slave'} at 127.0.0.1:55528
I20250903 14:23:06.553306 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "888cbb3e8e724c5f8b19bf129868cea9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
W20250903 14:23:06.554303 958 leader_election.cc:343] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Illegal state: must be running to vote when last-logged opid is not known
I20250903 14:23:06.557677 1710 leader_election.cc:290] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.560482 1700 tablet_bootstrap.cc:492] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:06.560665 1700 ts_tablet_manager.cc:1397] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.004s
I20250903 14:23:06.561403 1701 log.cc:1199] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/cbd068ad8e4448ba96e1bef32449a7c0
I20250903 14:23:06.557121 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "888cbb3e8e724c5f8b19bf129868cea9" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:06.561604 1700 raft_consensus.cc:357] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.561762 1298 raft_consensus.cc:3058] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.561839 1700 raft_consensus.cc:383] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.561939 1700 raft_consensus.cc:738] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.562422 1700 consensus_queue.cc:260] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.563000 1700 ts_tablet_manager.cc:1428] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 14:23:06.563737 1710 consensus_queue.cc:1035] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.564284 1698 ts_tablet_manager.cc:1918] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.564441 1698 log.cc:1199] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/cbd068ad8e4448ba96e1bef32449a7c0
I20250903 14:23:06.564942 1698 ts_tablet_manager.cc:1939] T cbd068ad8e4448ba96e1bef32449a7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:06.565308 1298 raft_consensus.cc:2466] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:06.565763 1660 raft_consensus.cc:491] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:06.565939 1660 raft_consensus.cc:513] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.566210 958 leader_election.cc:304] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters: 5d4ed06db54c4b5199e190b837bc53fe
I20250903 14:23:06.566563 1726 raft_consensus.cc:2802] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:06.566645 1660 leader_election.cc:290] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.567802 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a86369c9b3b4321abe742146a37623b" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:06.568130 1157 raft_consensus.cc:2466] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:06.569346 958 leader_election.cc:304] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:06.570008 1660 raft_consensus.cc:2802] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:06.570140 1660 raft_consensus.cc:491] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:06.570235 1660 raft_consensus.cc:3058] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.569264 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a86369c9b3b4321abe742146a37623b" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:06.572499 684 catalog_manager.cc:2149] created HMS catalog entry for table [id=ba26d86544084bfba8a647142ec79ed1]
W20250903 14:23:06.572542 958 leader_election.cc:343] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Illegal state: must be running to vote when last-logged opid is not known
I20250903 14:23:06.577893 1660 raft_consensus.cc:513] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.579157 1660 leader_election.cc:290] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.580535 1709 consensus_queue.cc:1035] T e6db03a8c88941278f0ab15599e4f975 P b03995cb49e94f108cf8caa7a502eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.581504 684 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46974:
table { table_name: "default.Table_eb580089aaa843ff90baf8a5795d8f56" } new_table_name: "default.Table_37cdaa8349a94cf1abcf2b79b208eeee" modify_external_catalogs: true
I20250903 14:23:06.552738 1706 tablet_bootstrap.cc:654] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.585520 1701 ts_tablet_manager.cc:1939] T cbd068ad8e4448ba96e1bef32449a7c0 P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:06.586822 1701 tablet_replica.cc:331] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:06.587035 678 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet cbd068ad8e4448ba96e1bef32449a7c0 (table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]) successfully deleted
I20250903 14:23:06.590724 1701 raft_consensus.cc:2241] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.591010 1701 raft_consensus.cc:2270] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.593132 1273 tablet_service.cc:1468] Processing CreateTablet for tablet 77c8726942e045ebac2610ab6b40e743 (DEFAULT_TABLE table=default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.593752 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c8726942e045ebac2610ab6b40e743. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.592177 1001 tablet_service.cc:1468] Processing CreateTablet for tablet 77c8726942e045ebac2610ab6b40e743 (DEFAULT_TABLE table=default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.594669 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c8726942e045ebac2610ab6b40e743. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.594841 1708 tablet_bootstrap.cc:654] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.598156 1726 raft_consensus.cc:695] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:06.598646 1726 consensus_queue.cc:237] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.601850 1298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a86369c9b3b4321abe742146a37623b" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:06.602934 1700 tablet_bootstrap.cc:492] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
W20250903 14:23:06.603746 958 leader_election.cc:343] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Illegal state: must be running to vote when last-logged opid is not known
I20250903 14:23:06.604213 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a86369c9b3b4321abe742146a37623b" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:06.604458 1158 raft_consensus.cc:3058] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.604511 1660 consensus_queue.cc:1035] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.605018 1700 tablet_bootstrap.cc:654] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.606045 686 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1] alter complete (version 1)
I20250903 14:23:06.607693 1136 tablet_service.cc:1468] Processing CreateTablet for tablet 77c8726942e045ebac2610ab6b40e743 (DEFAULT_TABLE table=default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.608431 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c8726942e045ebac2610ab6b40e743. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.610074 1698 tablet_replica.cc:331] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:06.610313 1698 raft_consensus.cc:2241] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.610899 678 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet cbd068ad8e4448ba96e1bef32449a7c0 (table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]) successfully deleted
I20250903 14:23:06.610951 1698 raft_consensus.cc:2270] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.611985 1698 ts_tablet_manager.cc:1905] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.615660 1660 consensus_queue.cc:1035] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.621377 686 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483] alter complete (version 1)
I20250903 14:23:06.625864 686 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46974:
table { table_name: "default.Table_2e6fe8bc133e4eb68510d15bd758709a" } new_table_name: "default.Table_54d50ef1cebc4d34a9a9521fb7b13929" modify_external_catalogs: true
I20250903 14:23:06.626282 1158 raft_consensus.cc:2466] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:06.628680 1707 ts_tablet_manager.cc:1918] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.628993 1707 log.cc:1199] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/0dc5829b0f9d4facb2d6da58c6be842f
I20250903 14:23:06.629623 1707 ts_tablet_manager.cc:1939] T 0dc5829b0f9d4facb2d6da58c6be842f P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:06.631827 1707 tablet_replica.cc:331] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:06.632170 1707 raft_consensus.cc:2241] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.632484 1707 raft_consensus.cc:2270] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.633620 1707 ts_tablet_manager.cc:1905] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.638940 1698 ts_tablet_manager.cc:1918] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:23:06.639266 1698 log.cc:1199] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/e6522648980345ed89a6657858b83d9a
I20250903 14:23:06.639930 1698 ts_tablet_manager.cc:1939] T e6522648980345ed89a6657858b83d9a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:06.641055 678 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet e6522648980345ed89a6657858b83d9a (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]) successfully deleted
I20250903 14:23:06.641925 1698 tablet_replica.cc:331] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:06.644495 958 leader_election.cc:304] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters: b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:06.645200 1743 raft_consensus.cc:2802] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:06.645401 1743 raft_consensus.cc:695] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:06.646001 1743 consensus_queue.cc:237] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.656351 1710 consensus_queue.cc:1035] T 9e61f697da744b0099dbd6939c1bc57c P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.591729 1701 ts_tablet_manager.cc:1905] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.693760 1706 tablet_bootstrap.cc:492] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:06.694025 1706 ts_tablet_manager.cc:1397] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.150s user 0.006s sys 0.000s
I20250903 14:23:06.695606 1706 raft_consensus.cc:357] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.696099 1706 raft_consensus.cc:383] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.696353 1706 raft_consensus.cc:738] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.696727 1706 consensus_queue.cc:260] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.697507 1706 ts_tablet_manager.cc:1428] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 14:23:06.700585 1700 tablet_bootstrap.cc:492] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:06.700804 1700 ts_tablet_manager.cc:1397] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.007s
I20250903 14:23:06.701833 1700 raft_consensus.cc:357] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.702158 1700 raft_consensus.cc:383] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.702273 1700 raft_consensus.cc:738] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.702612 1700 consensus_queue.cc:260] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.703406 1698 raft_consensus.cc:2241] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:06.704173 1698 pending_rounds.cc:70] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Trying to abort 1 pending ops.
I20250903 14:23:06.705204 1698 pending_rounds.cc:77] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196300847513640960 op_type: NO_OP noop_request { }
I20250903 14:23:06.705541 1698 raft_consensus.cc:2887] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 14:23:06.705730 1698 raft_consensus.cc:2270] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.704140 1700 ts_tablet_manager.cc:1428] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250903 14:23:06.725638 1701 ts_tablet_manager.cc:1918] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.726406 1698 ts_tablet_manager.cc:1905] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.734596 1698 ts_tablet_manager.cc:1918] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.734860 1698 log.cc:1199] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/888cbb3e8e724c5f8b19bf129868cea9
I20250903 14:23:06.735487 1698 ts_tablet_manager.cc:1939] T 888cbb3e8e724c5f8b19bf129868cea9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:06.740623 678 catalog_manager.cc:4928] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): tablet 888cbb3e8e724c5f8b19bf129868cea9 (table default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]) successfully deleted
I20250903 14:23:06.741137 1706 tablet_bootstrap.cc:492] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
W20250903 14:23:06.742642 678 catalog_manager.cc:4895] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): delete failed for tablet 0dc5829b0f9d4facb2d6da58c6be842f because the tablet was not found. No further retry: Not found: Tablet not found: 0dc5829b0f9d4facb2d6da58c6be842f
W20250903 14:23:06.743076 678 catalog_manager.cc:4895] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): delete failed for tablet 5a4cd5dd0f3243be9bba8defd097e2fe because the tablet was not found. No further retry: Not found: Tablet not found: 5a4cd5dd0f3243be9bba8defd097e2fe
I20250903 14:23:06.744065 1706 tablet_bootstrap.cc:654] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.745280 1701 log.cc:1199] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/5a4cd5dd0f3243be9bba8defd097e2fe
I20250903 14:23:06.745875 1701 ts_tablet_manager.cc:1939] T 5a4cd5dd0f3243be9bba8defd097e2fe P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:06.747370 678 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet 5a4cd5dd0f3243be9bba8defd097e2fe (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]) successfully deleted
I20250903 14:23:06.747946 1701 tablet_replica.cc:331] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:06.748220 1701 raft_consensus.cc:2241] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:06.748600 683 catalog_manager.cc:2149] created HMS catalog entry for table [id=2cd7578b26ee4765836cf4c316fdcfb3]
I20250903 14:23:06.748747 1701 raft_consensus.cc:2270] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.751230 1701 ts_tablet_manager.cc:1905] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.761783 758 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]
I20250903 14:23:06.762869 758 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39614 (request call id 33) took 3573 ms. Trace:
I20250903 14:23:06.763015 758 rpcz_store.cc:276] 0903 14:23:03.189825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.252391 (+2062566us) service_pool.cc:225] Handling call
0903 14:23:05.953935 (+701544us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 14:23:06.762801 (+808866us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":506868,"Hive Metastore.run_cpu_time_us":290,"Hive Metastore.run_wall_time_us":300480}
W20250903 14:23:06.763397 1095 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:39849, user_credentials={real_user=slave}} blocked reactor thread for 230249us
I20250903 14:23:06.767953 1749 raft_consensus.cc:491] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:06.768204 1749 raft_consensus.cc:513] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.768997 1749 leader_election.cc:290] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.770192 1204 heartbeater.cc:499] Master 127.0.104.62:37899 was elected leader, sending a full tablet report...
I20250903 14:23:06.771785 1020 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f7427e774b42b1bc3de1e70792f7c0" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:06.772038 1020 raft_consensus.cc:2466] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:06.772426 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f7427e774b42b1bc3de1e70792f7c0" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:06.773198 1296 raft_consensus.cc:2466] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:06.775139 1749 consensus_queue.cc:1035] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:06.775991 1095 leader_election.cc:304] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:06.779857 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu pid 660
I20250903 14:23:06.780474 1001 tablet_service.cc:1468] Processing CreateTablet for tablet 9a654351883e481496a2778b861fddfc (DEFAULT_TABLE table=default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.780982 1001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a654351883e481496a2778b861fddfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.781009 721 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 14:23:06.784749 1273 tablet_service.cc:1468] Processing CreateTablet for tablet 9a654351883e481496a2778b861fddfc (DEFAULT_TABLE table=default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.785380 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a654351883e481496a2778b861fddfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.790289 683 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]). Expected version 1 got 0
I20250903 14:23:06.777407 1737 raft_consensus.cc:2802] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:06.803900 1737 raft_consensus.cc:491] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:06.804057 1737 raft_consensus.cc:3058] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.780542 1136 tablet_service.cc:1468] Processing CreateTablet for tablet 9a654351883e481496a2778b861fddfc (DEFAULT_TABLE table=default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:06.807973 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a654351883e481496a2778b861fddfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:06.784781 1708 tablet_bootstrap.cc:492] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:06.809633 1708 ts_tablet_manager.cc:1397] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.289s user 0.006s sys 0.016s
I20250903 14:23:06.810539 1708 raft_consensus.cc:357] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.810745 1708 raft_consensus.cc:383] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.810832 1708 raft_consensus.cc:738] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.811079 1708 consensus_queue.cc:260] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.811738 1708 ts_tablet_manager.cc:1428] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 14:23:06.812134 1708 tablet_bootstrap.cc:492] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:06.814145 1708 tablet_bootstrap.cc:654] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.823457 1753 raft_consensus.cc:491] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:06.823654 1753 raft_consensus.cc:513] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.824362 1753 leader_election.cc:290] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.825505 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d67a811c1734c13b9119ed59e8ad1fb" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:06.825723 1296 raft_consensus.cc:2466] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:06.826025 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d67a811c1734c13b9119ed59e8ad1fb" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:06.826308 958 leader_election.cc:304] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:06.826867 1753 raft_consensus.cc:2802] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:06.827059 1753 raft_consensus.cc:491] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:06.827226 1753 raft_consensus.cc:3058] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.826282 1158 raft_consensus.cc:2466] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:06.807343 1737 raft_consensus.cc:513] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.830396 1737 leader_election.cc:290] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Requested vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.776088 1707 ts_tablet_manager.cc:1918] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.834292 1707 log.cc:1199] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/5a4cd5dd0f3243be9bba8defd097e2fe
I20250903 14:23:06.835170 1700 tablet_bootstrap.cc:492] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:06.835129 1020 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f7427e774b42b1bc3de1e70792f7c0" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:06.835348 1020 raft_consensus.cc:3058] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.835927 1707 ts_tablet_manager.cc:1939] T 5a4cd5dd0f3243be9bba8defd097e2fe P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:06.837240 1700 tablet_bootstrap.cc:654] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.838482 1020 raft_consensus.cc:2466] T f2f7427e774b42b1bc3de1e70792f7c0 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:06.838943 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f7427e774b42b1bc3de1e70792f7c0" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:06.839141 1296 raft_consensus.cc:3058] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.840487 1095 leader_election.cc:304] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:06.841225 1707 tablet_replica.cc:331] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:06.841459 1707 raft_consensus.cc:2241] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.841638 1707 raft_consensus.cc:2270] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.842024 1706 tablet_bootstrap.cc:492] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:06.842218 1706 ts_tablet_manager.cc:1397] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.000s
I20250903 14:23:06.842274 1707 ts_tablet_manager.cc:1905] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.842358 1753 raft_consensus.cc:513] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.842886 1296 raft_consensus.cc:2466] T f2f7427e774b42b1bc3de1e70792f7c0 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:06.843997 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d67a811c1734c13b9119ed59e8ad1fb" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:06.844327 1158 raft_consensus.cc:3058] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.844959 1759 raft_consensus.cc:2802] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:06.845240 1759 raft_consensus.cc:695] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:06.846522 1759 consensus_queue.cc:237] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.843304 1706 raft_consensus.cc:357] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.850062 1706 raft_consensus.cc:383] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.850479 1706 raft_consensus.cc:738] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.851045 1706 consensus_queue.cc:260] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.851572 1158 raft_consensus.cc:2466] T 9d67a811c1734c13b9119ed59e8ad1fb P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:06.852406 958 leader_election.cc:304] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:06.852800 1726 raft_consensus.cc:2802] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:06.852999 1726 raft_consensus.cc:695] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:06.853294 1726 consensus_queue.cc:237] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.855940 1706 ts_tablet_manager.cc:1428] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20250903 14:23:06.861173 1706 tablet_bootstrap.cc:492] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:06.862412 1700 tablet_bootstrap.cc:492] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:06.862722 1700 ts_tablet_manager.cc:1397] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.001s
I20250903 14:23:06.864138 1700 raft_consensus.cc:357] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.864500 1700 raft_consensus.cc:383] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.864735 1700 raft_consensus.cc:738] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.865163 1700 consensus_queue.cc:260] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.865693 1708 tablet_bootstrap.cc:492] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:06.865916 1708 ts_tablet_manager.cc:1397] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.001s
I20250903 14:23:06.866169 1700 ts_tablet_manager.cc:1428] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:06.865901 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d67a811c1734c13b9119ed59e8ad1fb" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:06.866649 1296 raft_consensus.cc:3058] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:06.866789 1708 raft_consensus.cc:357] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.866998 1708 raft_consensus.cc:383] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.867089 1708 raft_consensus.cc:738] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.867347 1708 consensus_queue.cc:260] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.868481 1708 ts_tablet_manager.cc:1428] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 14:23:06.868815 1708 tablet_bootstrap.cc:492] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:06.870752 1296 raft_consensus.cc:2466] T 9d67a811c1734c13b9119ed59e8ad1fb P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:06.871510 1708 tablet_bootstrap.cc:654] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.872669 759 catalog_manager.cc:2149] created HMS catalog entry for table [id=8649e9b9d8ea44429fd251a29586363f]
I20250903 14:23:06.873147 1706 tablet_bootstrap.cc:654] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
W20250903 14:23:06.873826 759 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica f4c58d929e53460883d05e0f14f20780 is not leader of this config: current role FOLLOWER
I20250903 14:23:06.876922 1707 ts_tablet_manager.cc:1918] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:06.879213 1753 leader_election.cc:290] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:06.880708 1701 ts_tablet_manager.cc:1918] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:23:06.884410 1707 log.cc:1199] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/cbd068ad8e4448ba96e1bef32449a7c0
I20250903 14:23:06.885388 1707 ts_tablet_manager.cc:1939] T cbd068ad8e4448ba96e1bef32449a7c0 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:06.905180 1706 tablet_bootstrap.cc:492] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:06.912108 1701 log.cc:1199] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/e6522648980345ed89a6657858b83d9a
I20250903 14:23:06.912240 1706 ts_tablet_manager.cc:1397] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.011s
I20250903 14:23:06.912734 1701 ts_tablet_manager.cc:1939] T e6522648980345ed89a6657858b83d9a P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:06.913268 1706 raft_consensus.cc:357] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.913527 1706 raft_consensus.cc:383] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.913604 1706 raft_consensus.cc:738] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.913846 1706 consensus_queue.cc:260] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.914371 1701 tablet_replica.cc:331] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:06.914477 1706 ts_tablet_manager.cc:1428] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 14:23:06.914631 1701 raft_consensus.cc:2241] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.914841 1701 raft_consensus.cc:2270] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.915577 1701 ts_tablet_manager.cc:1905] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.914846 1706 tablet_bootstrap.cc:492] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe: Bootstrap starting.
I20250903 14:23:06.929085 1706 tablet_bootstrap.cc:654] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.931620 1707 tablet_replica.cc:331] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:06.931946 1707 raft_consensus.cc:2241] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.932895 1707 raft_consensus.cc:2270] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.933451 1708 tablet_bootstrap.cc:492] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:06.933629 1708 ts_tablet_manager.cc:1397] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.006s
I20250903 14:23:06.934661 1708 raft_consensus.cc:357] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.934922 1708 raft_consensus.cc:383] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.935015 1708 raft_consensus.cc:738] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.935253 1708 consensus_queue.cc:260] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.935863 1708 ts_tablet_manager.cc:1428] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:23:06.936219 1708 tablet_bootstrap.cc:492] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:06.938511 1708 tablet_bootstrap.cc:654] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:06.940042 760 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=3379b0e151684210ac6845928dea7e14]
I20250903 14:23:06.941095 760 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 32) took 3858 ms. Trace:
I20250903 14:23:06.941234 760 rpcz_store.cc:276] 0903 14:23:03.082263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.133139 (+2050876us) service_pool.cc:225] Handling call
0903 14:23:05.952226 (+819087us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:05.952853 (+ 627us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:06.203415 (+250562us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:06.204206 (+ 791us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:06.204210 (+ 4us) write_op.cc:454] Released schema lock
0903 14:23:06.204438 (+ 228us) catalog_manager.cc:2156] Rolling back HMS table creation
0903 14:23:06.941045 (+736607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:06.204123 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica f4c58d929e53460883d05e0f14f20780 is not leader of this config: current role FOLLOWER)
0903 14:23:06.204138 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:06.204146 (+ 8us) write_op.cc:454] Released schema lock
0903 14:23:06.204151 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":819198,"Hive Metastore.run_cpu_time_us":516,"Hive Metastore.run_wall_time_us":166004,"child_traces":[["op",{}]]}
I20250903 14:23:06.944095 1707 ts_tablet_manager.cc:1905] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.944980 1706 tablet_bootstrap.cc:492] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe: No bootstrap required, opened a new log
I20250903 14:23:06.945155 1706 ts_tablet_manager.cc:1397] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.001s
I20250903 14:23:06.946079 1706 raft_consensus.cc:357] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.946285 1706 raft_consensus.cc:383] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.946388 1706 raft_consensus.cc:738] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03995cb49e94f108cf8caa7a502eafe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.946666 1706 consensus_queue.cc:260] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.949247 1706 ts_tablet_manager.cc:1428] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250903 14:23:06.961838 1707 ts_tablet_manager.cc:1918] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:06.966475 1707 log.cc:1199] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/888cbb3e8e724c5f8b19bf129868cea9
I20250903 14:23:06.967118 1707 ts_tablet_manager.cc:1939] T 888cbb3e8e724c5f8b19bf129868cea9 P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:06.972640 1701 ts_tablet_manager.cc:1918] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 14:23:06.973065 1707 tablet_replica.cc:331] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
I20250903 14:23:06.980069 1707 raft_consensus.cc:2241] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:06.980826 1707 raft_consensus.cc:2270] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:06.982369 1707 ts_tablet_manager.cc:1905] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:06.984788 1701 log.cc:1199] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/888cbb3e8e724c5f8b19bf129868cea9
I20250903 14:23:06.985673 1708 tablet_bootstrap.cc:492] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:06.985929 1708 ts_tablet_manager.cc:1397] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.001s
I20250903 14:23:06.986864 1708 raft_consensus.cc:357] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.987109 1708 raft_consensus.cc:383] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:06.987212 1708 raft_consensus.cc:738] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:06.987515 1708 consensus_queue.cc:260] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:06.988313 1708 ts_tablet_manager.cc:1428] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 14:23:06.988713 1708 tablet_bootstrap.cc:492] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe: Bootstrap starting.
I20250903 14:23:06.992887 1701 ts_tablet_manager.cc:1939] T 888cbb3e8e724c5f8b19bf129868cea9 P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:07.014962 1707 ts_tablet_manager.cc:1918] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 14:23:07.015197 1708 tablet_bootstrap.cc:654] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:07.016469 1707 log.cc:1199] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/e6522648980345ed89a6657858b83d9a
I20250903 14:23:07.016997 1707 ts_tablet_manager.cc:1939] T e6522648980345ed89a6657858b83d9a P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:07.032964 1708 tablet_bootstrap.cc:492] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe: No bootstrap required, opened a new log
I20250903 14:23:07.033335 1708 ts_tablet_manager.cc:1397] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.003s
I20250903 14:23:07.034690 1708 raft_consensus.cc:357] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.035079 1708 raft_consensus.cc:383] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:07.035233 1708 raft_consensus.cc:738] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Initialized, Role: FOLLOWER
I20250903 14:23:07.035595 1708 consensus_queue.cc:260] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.038527 1708 ts_tablet_manager.cc:1428] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 14:23:07.085882 759 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=8649e9b9d8ea44429fd251a29586363f]
I20250903 14:23:07.086330 759 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39614 (request call id 34) took 3888 ms. Trace:
I20250903 14:23:07.086428 759 rpcz_store.cc:276] 0903 14:23:03.197473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 14:23:05.304638 (+2107165us) service_pool.cc:225] Handling call
0903 14:23:05.954554 (+649916us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 14:23:05.955126 (+ 572us) catalog_manager.cc:2122] Created new table and tablet info
0903 14:23:06.872626 (+917500us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 14:23:06.873663 (+ 1037us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:06.873667 (+ 4us) write_op.cc:454] Released schema lock
0903 14:23:06.873955 (+ 288us) catalog_manager.cc:2156] Rolling back HMS table creation
0903 14:23:07.086258 (+212303us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 14:23:06.873582 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica f4c58d929e53460883d05e0f14f20780 is not leader of this config: current role FOLLOWER)
0903 14:23:06.873596 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
0903 14:23:06.873603 (+ 7us) write_op.cc:454] Released schema lock
0903 14:23:06.873608 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":871826,"Hive Metastore.run_cpu_time_us":585,"Hive Metastore.run_wall_time_us":256994,"child_traces":[["op",{}]]}
W20250903 14:23:07.087558 1354 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:37899)
I20250903 14:23:07.110512 678 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet 5a4cd5dd0f3243be9bba8defd097e2fe (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]) successfully deleted
I20250903 14:23:07.110818 678 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet cbd068ad8e4448ba96e1bef32449a7c0 (table default.table_4aee0c2e9c0b43938b9dc11b7c6bcd9d [id=829c730ca4ee4d2792c66bf244615417]) successfully deleted
I20250903 14:23:07.110989 678 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet 888cbb3e8e724c5f8b19bf129868cea9 (table default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]) successfully deleted
I20250903 14:23:07.110735 683 catalog_manager.cc:5582] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2). New cstate: current_term: 1 leader_uuid: "5d4ed06db54c4b5199e190b837bc53fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } }
I20250903 14:23:07.111234 678 catalog_manager.cc:4928] TS 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): tablet e6522648980345ed89a6657858b83d9a (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]) successfully deleted
I20250903 14:23:07.111586 678 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet e6522648980345ed89a6657858b83d9a (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]) successfully deleted
I20250903 14:23:07.111490 683 catalog_manager.cc:5582] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20250903 14:23:07.111819 678 catalog_manager.cc:4928] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): tablet 888cbb3e8e724c5f8b19bf129868cea9 (table default.table_293e76fad1c145eb963ab9e87c20ce43 [id=671393d4fe2249da97a8068fce39e09d]) successfully deleted
I20250903 14:23:07.111999 683 catalog_manager.cc:5674] TS 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849) does not have the latest schema for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]). Expected version 1 got 0
W20250903 14:23:07.112074 678 catalog_manager.cc:4895] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): delete failed for tablet b09ba744dc744fc3ac61383549c4babc because the tablet was not found. No further retry: Not found: Tablet not found: b09ba744dc744fc3ac61383549c4babc
W20250903 14:23:07.112321 678 catalog_manager.cc:4895] TS b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): delete failed for tablet 0dc5829b0f9d4facb2d6da58c6be842f because the tablet was not found. No further retry: Not found: Tablet not found: 0dc5829b0f9d4facb2d6da58c6be842f
I20250903 14:23:07.117288 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu with pid 660
W20250903 14:23:07.157516 1356 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:37899)
W20250903 14:23:07.158304 1356 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:37899)
W20250903 14:23:07.159155 1356 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:37899)
W20250903 14:23:07.163081 1527 connection.cc:537] server connection from 127.0.0.1:38064 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:07.166157 747 connection.cc:537] server connection from 127.0.0.1:45734 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:07.176748 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:07.158929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37899 (local address 127.0.104.3:35687)
0903 14:23:07.161501 (+ 2572us) negotiation.cc:107] Waiting for socket to connect
0903 14:23:07.161528 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 14:23:07.161685 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 14:23:07.175012 (+ 13327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2439,"thread_start_us":2862,"threads_started":1}
W20250903 14:23:07.180245 1775 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:07.163870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37899 (local address 127.0.0.1:46998)
0903 14:23:07.178043 (+ 14173us) negotiation.cc:107] Waiting for socket to connect
0903 14:23:07.178284 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":14025,"thread_start_us":13895,"threads_started":1}
I20250903 14:23:07.182179 1759 raft_consensus.cc:491] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:07.182394 1759 raft_consensus.cc:513] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.183077 1759 leader_election.cc:290] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:07.183954 1020 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a654351883e481496a2778b861fddfc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:07.183983 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a654351883e481496a2778b861fddfc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:07.184186 1020 raft_consensus.cc:2466] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:07.184201 1296 raft_consensus.cc:2466] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 0.
I20250903 14:23:07.184675 1095 leader_election.cc:304] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:07.185040 1759 raft_consensus.cc:2802] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:07.185153 1759 raft_consensus.cc:491] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:07.185253 1759 raft_consensus.cc:3058] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.189504 1759 raft_consensus.cc:513] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.190495 1759 leader_election.cc:290] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Requested vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:07.191108 1020 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a654351883e481496a2778b861fddfc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd"
I20250903 14:23:07.191313 1020 raft_consensus.cc:3058] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.191224 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a654351883e481496a2778b861fddfc" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:07.191540 1296 raft_consensus.cc:3058] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.194195 1020 raft_consensus.cc:2466] T 9a654351883e481496a2778b861fddfc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:07.194777 1753 raft_consensus.cc:491] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:07.194916 1753 raft_consensus.cc:513] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.198952 1296 raft_consensus.cc:2466] T 9a654351883e481496a2778b861fddfc P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
I20250903 14:23:07.201010 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e78a7c3701f464080b7b887574a6b7a" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:07.201376 1158 raft_consensus.cc:2466] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:07.201967 958 leader_election.cc:304] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:07.202358 1726 raft_consensus.cc:2802] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:07.202461 1726 raft_consensus.cc:491] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:07.202560 1726 raft_consensus.cc:3058] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.205614 1726 raft_consensus.cc:513] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.206507 1726 leader_election.cc:290] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:07.207039 1095 leader_election.cc:304] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters:
I20250903 14:23:07.207110 1753 leader_election.cc:290] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:07.207551 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e78a7c3701f464080b7b887574a6b7a" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:07.207779 1158 raft_consensus.cc:3058] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.207909 1759 raft_consensus.cc:2802] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:07.209146 1297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e78a7c3701f464080b7b887574a6b7a" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:07.209398 1297 raft_consensus.cc:2466] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:07.210317 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e78a7c3701f464080b7b887574a6b7a" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:07.210577 1296 raft_consensus.cc:3058] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.211359 1759 raft_consensus.cc:695] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:07.213083 1759 consensus_queue.cc:237] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.210987 1158 raft_consensus.cc:2466] T 0e78a7c3701f464080b7b887574a6b7a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:07.213930 958 leader_election.cc:304] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:07.214365 1753 raft_consensus.cc:2802] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:07.214561 1753 raft_consensus.cc:695] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:07.214838 1753 consensus_queue.cc:237] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.215116 1296 raft_consensus.cc:2466] T 0e78a7c3701f464080b7b887574a6b7a P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:07.253571 1496 raft_consensus.cc:491] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:07.253813 1496 raft_consensus.cc:513] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.254608 1496 leader_election.cc:290] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:07.255221 1020 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a86369c9b3b4321abe742146a37623b" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" is_pre_election: true
I20250903 14:23:07.255270 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a86369c9b3b4321abe742146a37623b" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:07.255600 1158 raft_consensus.cc:2391] T 3a86369c9b3b4321abe742146a37623b P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b03995cb49e94f108cf8caa7a502eafe in current term 1: Already voted for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in this term.
I20250903 14:23:07.256014 1234 leader_election.cc:304] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b03995cb49e94f108cf8caa7a502eafe; no voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe
I20250903 14:23:07.256346 1496 raft_consensus.cc:3058] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.258330 1496 raft_consensus.cc:2747] T 3a86369c9b3b4321abe742146a37623b P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:07.386317 1753 raft_consensus.cc:491] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:07.386591 1753 raft_consensus.cc:513] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.387538 1753 leader_election.cc:290] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:07.388177 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c8726942e045ebac2610ab6b40e743" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
I20250903 14:23:07.388269 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c8726942e045ebac2610ab6b40e743" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:07.388417 1158 raft_consensus.cc:2466] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:07.388531 1296 raft_consensus.cc:2466] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 0.
I20250903 14:23:07.388898 958 leader_election.cc:304] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:07.389199 1753 raft_consensus.cc:2802] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 14:23:07.389336 1753 raft_consensus.cc:491] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:07.389442 1753 raft_consensus.cc:3058] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.392958 1753 raft_consensus.cc:513] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.394266 1158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c8726942e045ebac2610ab6b40e743" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe"
I20250903 14:23:07.394515 1158 raft_consensus.cc:3058] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.394500 1753 leader_election.cc:290] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Requested vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:07.396025 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c8726942e045ebac2610ab6b40e743" candidate_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:07.396274 1296 raft_consensus.cc:3058] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe [term 0 FOLLOWER]: Advancing to term 1
I20250903 14:23:07.397007 1158 raft_consensus.cc:2466] T 77c8726942e045ebac2610ab6b40e743 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:07.397456 958 leader_election.cc:304] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe; no voters:
I20250903 14:23:07.397733 1753 raft_consensus.cc:2802] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 14:23:07.397876 1753 raft_consensus.cc:695] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Running, Role: LEADER
I20250903 14:23:07.398213 1753 consensus_queue.cc:237] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:07.399706 1296 raft_consensus.cc:2466] T 77c8726942e045ebac2610ab6b40e743 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adb02d7e9fc4548aa7fd31323ff8ecd in term 1.
I20250903 14:23:07.618434 1783 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2e83636ce3ca47f3bfc9f84f6755eec2)
I20250903 14:23:07.618659 1783 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:07.619503 1784 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2e83636ce3ca47f3bfc9f84f6755eec2)
I20250903 14:23:07.619715 1783 leader_election.cc:290] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437)
I20250903 14:23:07.620018 1784 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:07.620333 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 3 candidate_status { last_received { term: 2 index: 80 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:07.620590 771 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bbe08ab62bf4073825bf1b598ad2edb in term 2.
I20250903 14:23:07.620877 1784 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:07.621328 1525 leader_election.cc:304] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780; no voters:
W20250903 14:23:07.621362 1527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:07.621644 1783 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250903 14:23:07.621728 748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20250903 14:23:07.621762 1783 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2e83636ce3ca47f3bfc9f84f6755eec2)
I20250903 14:23:07.621929 1783 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 2 FOLLOWER]: Advancing to term 3
I20250903 14:23:07.622529 1549 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 3 candidate_status { last_received { term: 2 index: 80 } } ignore_live_leader: false dest_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" is_pre_election: true
W20250903 14:23:07.623786 1527 leader_election.cc:336] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:07.623940 748 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:07.624871 1783 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:07.625329 1549 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4c58d929e53460883d05e0f14f20780 in current term 3: Already voted for candidate 2bbe08ab62bf4073825bf1b598ad2edb in this term.
I20250903 14:23:07.625555 1783 leader_election.cc:290] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 3 election: Requested vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437)
I20250903 14:23:07.626106 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" candidate_term: 3 candidate_status { last_received { term: 2 index: 80 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780"
I20250903 14:23:07.626230 746 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:07.626343 771 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 2 FOLLOWER]: Advancing to term 3
W20250903 14:23:07.627096 1527 leader_election.cc:336] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:07.628486 771 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bbe08ab62bf4073825bf1b598ad2edb in term 3.
I20250903 14:23:07.628708 1784 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250903 14:23:07.628950 1525 leader_election.cc:304] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780; no voters: 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:07.629344 1783 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 3 FOLLOWER]: Leader election won for term 3
I20250903 14:23:07.629760 1783 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [term 3 LEADER]: Becoming Leader. State: Replica: 2bbe08ab62bf4073825bf1b598ad2edb, State: Running, Role: LEADER
I20250903 14:23:07.630208 1783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 2.80, Last appended by leader: 80, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:07.631620 1789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbe08ab62bf4073825bf1b598ad2edb. Latest consensus state: current_term: 3 leader_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:07.631908 1789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:07.643963 1753 consensus_queue.cc:1035] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:07.652395 1753 consensus_queue.cc:1035] T 3a86369c9b3b4321abe742146a37623b P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:07.845598 1759 consensus_queue.cc:1035] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:07.852454 1759 consensus_queue.cc:1035] T f2f7427e774b42b1bc3de1e70792f7c0 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:07.963550 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--webserver_interface=127.0.104.62
--webserver_port=33279
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 14:23:07.999948 1753 consensus_queue.cc:1035] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:08.019752 1726 consensus_queue.cc:1035] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:08.020259 1726 consensus_queue.cc:1035] T 9d67a811c1734c13b9119ed59e8ad1fb P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 14:23:08.026784 1527 consensus_peers.cc:489] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb -> Peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Couldn't send request to peer 2e83636ce3ca47f3bfc9f84f6755eec2. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 14:23:08.032312 1726 consensus_queue.cc:1035] T 0e78a7c3701f464080b7b887574a6b7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:08.155336 771 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Refusing update from remote peer 2bbe08ab62bf4073825bf1b598ad2edb: Log matching property violated. Preceding OpId in replica: term: 2 index: 80. Preceding OpId from leader: term: 3 index: 81. (index mismatch)
I20250903 14:23:08.156252 1789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20250903 14:23:08.158840 1807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbe08ab62bf4073825bf1b598ad2edb. Latest consensus state: current_term: 3 leader_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:08.159279 1807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:08.163371 1789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:08.163668 1789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:08.165467 1807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:08.165714 1807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:08.212636 1798 consensus_queue.cc:1035] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:08.221275 1799 consensus_queue.cc:1035] T 9a654351883e481496a2778b861fddfc P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:08.278800 1790 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:23:08.281955 1790 catalog_manager.cc:671] Loaded metadata for table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]
I20250903 14:23:08.282580 1790 catalog_manager.cc:671] Loaded metadata for table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483]
I20250903 14:23:08.282989 1790 catalog_manager.cc:671] Loaded metadata for table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]
I20250903 14:23:08.283368 1790 catalog_manager.cc:671] Loaded metadata for table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3]
I20250903 14:23:08.283938 1790 catalog_manager.cc:671] Loaded metadata for table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c]
I20250903 14:23:08.284363 1790 catalog_manager.cc:671] Loaded metadata for table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]
I20250903 14:23:08.284924 1790 catalog_manager.cc:671] Loaded metadata for table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1]
I20250903 14:23:08.285277 1790 catalog_manager.cc:671] Loaded metadata for table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]
I20250903 14:23:08.286032 1790 catalog_manager.cc:671] Loaded metadata for table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]
I20250903 14:23:08.286473 1790 catalog_manager.cc:671] Loaded metadata for table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]
I20250903 14:23:08.286846 1790 catalog_manager.cc:671] Loaded metadata for table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1]
I20250903 14:23:08.287618 1790 catalog_manager.cc:671] Loaded metadata for table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]
I20250903 14:23:08.288144 1790 catalog_manager.cc:671] Loaded metadata for table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]
I20250903 14:23:08.288638 1790 catalog_manager.cc:671] Loaded metadata for table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]
I20250903 14:23:08.289144 1790 catalog_manager.cc:671] Loaded metadata for table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]
I20250903 14:23:08.291481 1782 consensus_queue.cc:1035] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:08.292335 1790 tablet_loader.cc:96] loaded metadata for tablet 05b219b49aa24cbe995c03d1098875dc (table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2])
I20250903 14:23:08.293452 1790 tablet_loader.cc:96] loaded metadata for tablet 0e78a7c3701f464080b7b887574a6b7a (table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c])
I20250903 14:23:08.294533 1790 tablet_loader.cc:96] loaded metadata for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13])
I20250903 14:23:08.295871 1790 tablet_loader.cc:96] loaded metadata for tablet 77c8726942e045ebac2610ab6b40e743 (table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1])
I20250903 14:23:08.298136 1790 tablet_loader.cc:96] loaded metadata for tablet 8ec2abff20f1413fad17fd393d8fd1e9 (table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63])
I20250903 14:23:08.299221 1790 tablet_loader.cc:96] loaded metadata for tablet 9a654351883e481496a2778b861fddfc (table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3])
I20250903 14:23:08.299938 1790 tablet_loader.cc:96] loaded metadata for tablet 9d67a811c1734c13b9119ed59e8ad1fb (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b])
I20250903 14:23:08.300778 1790 tablet_loader.cc:96] loaded metadata for tablet 9e61f697da744b0099dbd6939c1bc57c (table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b])
I20250903 14:23:08.302265 1790 tablet_loader.cc:96] loaded metadata for tablet b5afc9676cca4a9ba1841546f5d38d7a (table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9])
I20250903 14:23:08.302755 1790 tablet_loader.cc:96] loaded metadata for tablet c51713a4e7554285ba265825b8651efe (table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1])
I20250903 14:23:08.303836 1790 tablet_loader.cc:96] loaded metadata for tablet d1fcf42e749a463c8f47e275c0fe3439 (table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483])
I20250903 14:23:08.305316 1790 tablet_loader.cc:96] loaded metadata for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e])
I20250903 14:23:08.305377 1753 consensus_queue.cc:1035] T 77c8726942e045ebac2610ab6b40e743 P 1adb02d7e9fc4548aa7fd31323ff8ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 14:23:08.306612 1790 tablet_loader.cc:96] loaded metadata for tablet e6db03a8c88941278f0ab15599e4f975 (table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5])
I20250903 14:23:08.307114 1790 tablet_loader.cc:96] loaded metadata for tablet efadfb3a6ed5403e9d17f47891ac2206 (table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d])
I20250903 14:23:08.307641 1790 tablet_loader.cc:96] loaded metadata for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328])
I20250903 14:23:08.308183 1790 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:23:08.309790 1790 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:08.309903 1790 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:23:08.312115 1790 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:23:08.315773 1790 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Loaded TSK: 0
I20250903 14:23:08.318316 1790 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:23:08.320896 1790 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 14:23:08.322680 1790 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250903 14:23:08.372649 1533 master_service.cc:948] ReplaceTablet: received request to replace tablet e6522648980345ed89a6657858b83d9a from {username='slave'} at 127.0.0.1:38070
I20250903 14:23:08.373991 1531 master_service.cc:948] ReplaceTablet: received request to replace tablet 05b219b49aa24cbe995c03d1098875dc from {username='slave'} at 127.0.0.1:38070
I20250903 14:23:08.374107 1533 master_service.cc:948] ReplaceTablet: received request to replace tablet 8ec2abff20f1413fad17fd393d8fd1e9 from {username='slave'} at 127.0.0.1:38070
I20250903 14:23:08.370587 1539 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_8eb8eda2d5fd408a87a960fe890af54c" } modify_external_catalogs: true
I20250903 14:23:08.373817 1532 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_d7a178c8bc1546dd9464cf497775d66a" } modify_external_catalogs: true
I20250903 14:23:08.371369 1537 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38070:
name: "default.Table_32a70b2f108542938b8647dc4595cb31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:08.377887 1539 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_8eb8eda2d5fd408a87a960fe890af54c" } modify_external_catalogs: true
I20250903 14:23:08.377960 1532 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_d7a178c8bc1546dd9464cf497775d66a" } modify_external_catalogs: true
I20250903 14:23:08.371821 1536 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_152f6fe6ff314070b2b909c120bf6a10" } modify_external_catalogs: true
I20250903 14:23:08.371285 1534 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38070:
name: "default.Table_dd32253e3975467098b9075da837375c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:08.378455 1536 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_152f6fe6ff314070b2b909c120bf6a10" } modify_external_catalogs: true
I20250903 14:23:08.372524 1535 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38070:
name: "default.Table_e3decc796a014477b22f229bd0211387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:08.370231 1538 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38070:
name: "default.Table_663b9a72bf8543eb90c51bd282a2db4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:08.372011 1540 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38070:
name: "default.Table_0ba72da4f2b84fb39612871a39e8d19f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:08.387391 1533 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Sending DeleteTablet for 3 replicas of tablet 8ec2abff20f1413fad17fd393d8fd1e9
F20250903 14:23:08.388487 1616 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 28, previous 28; dumping first 7 out of 10 received events: NotificationEvent(eventId=23, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_152f6fe6ff314070b2b909c120bf6a10, message={"server":"","servicePrincipal":"","db":"default","table":"table_152f6fe6ff314070b2b909c120bf6a10","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_152f6fe6ff314070b2b909c120bf6a10\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_152f6fe6ff314070b2b909c120bf6a10\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_152f6fe6ff314070b2b909c120bf6a10\",\"kudu.table_id\":\"4ad47717c2424f57a3235322430e8d3c\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_219a289c9b7a44af91e47feece52778a, message={"server":"","servicePrincipal":"","db":"default","table":"table_219a289c9b7a44af91e47feece52778a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_219a289c9b7a44af91e47feece52778a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_219a289c9b7a44af91e47feece52778a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_219a289c9b7a44af91e47feece52778a\",\"kudu.table_id\":\"03d2198ad57f4f729123cf099287a61d\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_a8c65bc030834131a6a9b071e7cae292, message={"server":"","servicePrincipal":"","db":"default","table":"table_a8c65bc030834131a6a9b071e7cae292","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a8c65bc030834131a6a9b071e7cae292\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_a8c65bc030834131a6a9b071e7cae292\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a8c65bc030834131a6a9b071e7cae292\",\"kudu.table_id\":\"ba26d86544084bfba8a647142ec79ed1\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_6fd4c0ea3f684a06b60c606e942d3961, message={"server":"","servicePrincipal":"","db":"default","table":"table_6fd4c0ea3f684a06b60c606e942d3961","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6fd4c0ea3f684a06b60c606e942d3961\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_6fd4c0ea3f684a06b60c606e942d3961\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6fd4c0ea3f684a06b60c606e942d3961\",\"kudu.table_id\":\"2cd7578b26ee4765836cf4c316fdcfb3\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_d5a7e44d7d1c4005aa2bc39c855d2418, message={"server":"","servicePrincipal":"","db":"default","table":"table_d5a7e44d7d1c4005aa2bc39c855d2418","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d5a7e44d7d1c4005aa2bc39c855d2418\",\"kudu.table_id\":\"d35e7f78eeb345df9ad80071d3b6ccb9\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_0ba72da4f2b84fb39612871a39e8d19f, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba72da4f2b84fb39612871a39e8d19f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba72da4f2b84fb39612871a39e8d19f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_0ba72da4f2b84fb39612871a39e8d19f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba72da4f2b84fb39612871a39e8d19f\",\"kudu.table_id\":\"8649e9b9d8ea44429fd251a29586363f\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_d63d2c9adbc5421f9a1c01f637efe55b, message={"server":"","servicePrincipal":"","db":"default","table":"table_d63d2c9adbc5421f9a1c01f637efe55b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d63d2c9adbc5421f9a1c01f637efe55b\",\"kudu.table_id\":\"6f5c11cb07914fb299328948ea5a0516\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fc45c926dcd google::LogMessage::Fail() at ??:0
@ 0x7fc45c92ab93 google::LogMessage::SendToLog() at ??:0
I20250903 14:23:08.394673 1533 catalog_manager.cc:6652] ReplaceTablet: tablet 8ec2abff20f1413fad17fd393d8fd1e9 deleted and replaced by tablet f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:08.395208 1533 master_service.cc:948] ReplaceTablet: received request to replace tablet 05b219b49aa24cbe995c03d1098875dc from {username='slave'} at 127.0.0.1:38070
@ 0x7fc45c9267cc google::LogMessage::Flush() at ??:0
I20250903 14:23:08.405143 1531 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2bbe08ab62bf4073825bf1b598ad2edb: Sending DeleteTablet for 3 replicas of tablet 05b219b49aa24cbe995c03d1098875dc
I20250903 14:23:08.405717 1531 catalog_manager.cc:6652] ReplaceTablet: tablet 05b219b49aa24cbe995c03d1098875dc deleted and replaced by tablet 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:08.406100 1531 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_3368c82b8e6b4b149e8ad8d925c3d0e0" } modify_external_catalogs: true
I20250903 14:23:08.406378 1531 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38070:
table { table_name: "default.Table_3368c82b8e6b4b149e8ad8d925c3d0e0" } modify_external_catalogs: true
I20250903 14:23:08.407014 1533 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38070:
name: "default.Table_d63d2c9adbc5421f9a1c01f637efe55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fc45c926a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fc472d697b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fc472d68061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fc472d6fcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fc472d6fb59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fc45f68f2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc462ab36db start_thread at ??:0
@ 0x7fc45b7d771f clone at ??:0
I20250903 14:23:08.472807 1001 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ec2abff20f1413fad17fd393d8fd1e9 with delete_type TABLET_DATA_DELETED (replaced by tablet f9c4e2c45d19494f86c3c559d3d91202) from {username='slave'} at 127.0.0.1:55538
I20250903 14:23:08.473053 999 tablet_service.cc:1515] Processing DeleteTablet for tablet 05b219b49aa24cbe995c03d1098875dc with delete_type TABLET_DATA_DELETED (replaced by tablet 29e447cc964646169fbe6ac64090b63e) from {username='slave'} at 127.0.0.1:55538
I20250903 14:23:08.473460 1000 tablet_service.cc:1468] Processing CreateTablet for tablet 29e447cc964646169fbe6ac64090b63e (DEFAULT_TABLE table=default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:08.473548 1002 tablet_service.cc:1468] Processing CreateTablet for tablet f9c4e2c45d19494f86c3c559d3d91202 (DEFAULT_TABLE table=default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 14:23:08.473990 1000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29e447cc964646169fbe6ac64090b63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:08.474316 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9c4e2c45d19494f86c3c559d3d91202. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 14:23:08.477501 1826 tablet_replica.cc:331] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:08.477761 1826 raft_consensus.cc:2241] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:08.478271 1826 raft_consensus.cc:2270] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:08.479076 1826 ts_tablet_manager.cc:1905] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:08.488655 1826 ts_tablet_manager.cc:1918] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:08.489284 1826 log.cc:1199] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/8ec2abff20f1413fad17fd393d8fd1e9
I20250903 14:23:08.489924 1826 ts_tablet_manager.cc:1939] T 8ec2abff20f1413fad17fd393d8fd1e9 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:08.491531 1826 tablet_replica.cc:331] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:08.491606 1827 tablet_bootstrap.cc:492] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:08.491834 1826 raft_consensus.cc:2241] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:08.492924 1826 raft_consensus.cc:2270] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:08.494005 1826 ts_tablet_manager.cc:1905] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:08.494601 1827 tablet_bootstrap.cc:654] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
I20250903 14:23:08.500450 1826 ts_tablet_manager.cc:1918] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:08.500711 1826 log.cc:1199] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/05b219b49aa24cbe995c03d1098875dc
I20250903 14:23:08.501214 1826 ts_tablet_manager.cc:1939] T 05b219b49aa24cbe995c03d1098875dc P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:08.505056 1827 tablet_bootstrap.cc:492] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:08.505240 1827 ts_tablet_manager.cc:1397] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20250903 14:23:08.506261 1827 raft_consensus.cc:357] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:08.506516 1827 raft_consensus.cc:383] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:08.506614 1827 raft_consensus.cc:738] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:08.506879 1827 consensus_queue.cc:260] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:08.508225 1827 ts_tablet_manager.cc:1428] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 14:23:08.508580 1827 tablet_bootstrap.cc:492] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Bootstrap starting.
I20250903 14:23:08.510854 1827 tablet_bootstrap.cc:654] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Neither blocks nor log segments found. Creating new log.
W20250903 14:23:08.516615 1801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:08.517185 1801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:08.517374 1801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250903 14:23:08.517447 1827 tablet_bootstrap.cc:492] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd: No bootstrap required, opened a new log
I20250903 14:23:08.517621 1827 ts_tablet_manager.cc:1397] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20250903 14:23:08.518415 1827 raft_consensus.cc:357] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:08.518639 1827 raft_consensus.cc:383] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 14:23:08.518738 1827 raft_consensus.cc:738] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adb02d7e9fc4548aa7fd31323ff8ecd, State: Initialized, Role: FOLLOWER
I20250903 14:23:08.519012 1827 consensus_queue.cc:260] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:08.523687 1827 ts_tablet_manager.cc:1428] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
W20250903 14:23:08.533768 1801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:08.533903 1801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:08.534003 1801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:08.534073 1801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:08.553294 1801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=33279
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:08.556257 1801 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:08.560292 1801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:08.573771 1840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:08.575366 1841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:08.585489 1801 server_base.cc:1047] running on GCE node
W20250903 14:23:08.585479 1844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:08.586973 1801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:08.588864 1801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:08.590310 1801 hybrid_clock.cc:648] HybridClock initialized: now 1756909388590227 us; error 81 us; skew 500 ppm
I20250903 14:23:08.590809 1801 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:08.594273 1801 webserver.cc:480] Webserver started at http://127.0.104.62:33279/ using document root <none> and password file <none>
I20250903 14:23:08.595635 1801 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:08.595888 1801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:08.605353 1801 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20250903 14:23:08.610287 1852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:08.612910 1801 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250903 14:23:08.613206 1801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
format_stamp: "Formatted at 2025-09-03 14:22:58 on dist-test-slave-rxtz"
I20250903 14:23:08.614308 1801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:08.631162 1753 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:08.631413 1753 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:08.632169 1753 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:08.633397 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:08.633744 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:08.633912 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:08.634275 1753 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:08.637970 1801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:08.639168 1801 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:08.639585 1801 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250903 14:23:08.646276 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 1: this message will repeat every 5th retry.
I20250903 14:23:08.665424 1903 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37899 every 8 connection(s)
I20250903 14:23:08.665433 1801 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37899
I20250903 14:23:08.668231 1801 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
I20250903 14:23:08.677650 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 1801
I20250903 14:23:08.680708 1904 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:08.685195 1904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap starting.
I20250903 14:23:08.713032 1904 log.cc:826] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:08.715734 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:08.716039 1204 heartbeater.cc:499] Master 127.0.104.62:37899 was elected leader, sending a full tablet report...
I20250903 14:23:08.799654 1067 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:08.806579 1904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=79 ignored=0} inserts{seen=54 ignored=0} mutations{seen=61 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250903 14:23:08.807293 1904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap complete.
I20250903 14:23:08.812956 1904 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:08.815871 1904 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e83636ce3ca47f3bfc9f84f6755eec2, State: Initialized, Role: FOLLOWER
I20250903 14:23:08.816845 1904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 2.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:08.819593 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:08.820084 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:08.821198 1904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: configured and running, proceeding with master startup.
W20250903 14:23:08.827365 958 connection.cc:537] server connection from 127.0.0.1:55538 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:08.827874 1824 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:08.440059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.0.104.3:43373)
0903 14:23:08.467599 (+ 27540us) server_negotiation.cc:207] Beginning negotiation
0903 14:23:08.467611 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0903 14:23:08.473289 (+ 5678us) server_negotiation.cc:408] Connection header received
0903 14:23:08.473408 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:23:08.473414 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:23:08.473527 (+ 113us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:23:08.473655 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:23:08.827471 (+353816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58724: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":27162,"thread_start_us":27398,"threads_started":1}
W20250903 14:23:08.827920 1354 connection.cc:537] client connection to 127.0.104.60:43143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:08.828748 1354 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.60:43143)
W20250903 14:23:08.829133 1819 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:08.396045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.0.104.2:46863)
0903 14:23:08.417274 (+ 21229us) server_negotiation.cc:207] Beginning negotiation
0903 14:23:08.417285 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 14:23:08.417325 (+ 40us) server_negotiation.cc:408] Connection header received
0903 14:23:08.417441 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:23:08.417452 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:23:08.417599 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:23:08.417791 (+ 192us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:23:08.420943 (+ 3152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 14:23:08.474533 (+ 53590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 14:23:08.828034 (+353501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42560: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":20989,"thread_start_us":22794,"threads_started":1}
W20250903 14:23:08.829218 1354 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.60:43143)
W20250903 14:23:08.829926 1354 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.60:43143)
W20250903 14:23:08.830137 1354 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.60:43143)
W20250903 14:23:08.838184 1093 connection.cc:537] client connection to 127.0.104.60:43143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:08.839032 1232 connection.cc:537] client connection to 127.0.104.60:43143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:08.840021 956 connection.cc:537] client connection to 127.0.104.60:43143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:08.841975 1775 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:08.833577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43143 (local address 127.0.0.1:38086)
0903 14:23:08.841567 (+ 7990us) negotiation.cc:107] Waiting for socket to connect
0903 14:23:08.841664 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":7873,"threads_started":1}
I20250903 14:23:08.842671 1343 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:08.871943 1926 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:08.872190 1926 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: loading cluster ID for follower catalog manager: success
I20250903 14:23:08.876904 1926 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: acquiring CA information for follower catalog manager: success
I20250903 14:23:08.881094 1926 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:08.888522 1801 master_runner.cc:186] Error getting master registration for 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111),
I20250903 14:23:08.893955 1801 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:08.941573 1931 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2bbe08ab62bf4073825bf1b598ad2edb)
I20250903 14:23:08.941802 1931 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:08.944216 1931 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:08.946028 746 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:08.951021 1879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 4 candidate_status { last_received { term: 3 index: 85 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" is_pre_election: true
I20250903 14:23:08.951799 1879 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 2.
I20250903 14:23:08.952831 748 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb
I20250903 14:23:08.953356 1931 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250903 14:23:08.953474 1931 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2bbe08ab62bf4073825bf1b598ad2edb)
I20250903 14:23:08.953655 1931 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 3 FOLLOWER]: Advancing to term 4
I20250903 14:23:08.955999 1931 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:08.956588 1931 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 4 election: Requested vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:08.957222 1879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 4 candidate_status { last_received { term: 3 index: 85 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
I20250903 14:23:08.957470 1879 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 2 FOLLOWER]: Advancing to term 4
W20250903 14:23:08.958412 746 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:08.963608 1879 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 4.
I20250903 14:23:08.964358 748 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb
I20250903 14:23:08.964807 1931 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 4 FOLLOWER]: Leader election won for term 4
I20250903 14:23:08.965343 1931 raft_consensus.cc:695] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 4 LEADER]: Becoming Leader. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Running, Role: LEADER
I20250903 14:23:08.965680 1931 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 3.85, Last appended by leader: 85, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:08.967003 1935 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 4 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:08.967396 1935 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:08.968400 1936 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:23:08.972630 1936 catalog_manager.cc:671] Loaded metadata for table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]
I20250903 14:23:08.973263 1936 catalog_manager.cc:671] Loaded metadata for table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483]
I20250903 14:23:08.973721 1936 catalog_manager.cc:671] Loaded metadata for table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]
I20250903 14:23:08.974224 1936 catalog_manager.cc:671] Loaded metadata for table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3]
I20250903 14:23:08.974679 1936 catalog_manager.cc:671] Loaded metadata for table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c]
I20250903 14:23:08.975126 1936 catalog_manager.cc:671] Loaded metadata for table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]
I20250903 14:23:08.975970 1936 catalog_manager.cc:671] Loaded metadata for table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1]
I20250903 14:23:08.976418 1936 catalog_manager.cc:671] Loaded metadata for table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]
I20250903 14:23:08.977353 1936 catalog_manager.cc:671] Loaded metadata for table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]
I20250903 14:23:08.978039 1936 catalog_manager.cc:671] Loaded metadata for table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]
I20250903 14:23:08.978514 1936 catalog_manager.cc:671] Loaded metadata for table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1]
I20250903 14:23:08.979022 1936 catalog_manager.cc:671] Loaded metadata for table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]
I20250903 14:23:08.979422 1936 catalog_manager.cc:671] Loaded metadata for table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]
I20250903 14:23:08.979938 1936 catalog_manager.cc:671] Loaded metadata for table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]
I20250903 14:23:08.980396 1936 catalog_manager.cc:671] Loaded metadata for table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]
I20250903 14:23:08.983595 1936 tablet_loader.cc:96] loaded metadata for tablet 0e78a7c3701f464080b7b887574a6b7a (table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c])
I20250903 14:23:08.984071 1936 tablet_loader.cc:96] loaded metadata for tablet 29e447cc964646169fbe6ac64090b63e (table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2])
I20250903 14:23:08.984903 1936 tablet_loader.cc:96] loaded metadata for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13])
I20250903 14:23:08.986227 1936 tablet_loader.cc:96] loaded metadata for tablet 77c8726942e045ebac2610ab6b40e743 (table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1])
I20250903 14:23:08.987562 1936 tablet_loader.cc:96] loaded metadata for tablet 9a654351883e481496a2778b861fddfc (table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3])
I20250903 14:23:08.988454 1936 tablet_loader.cc:96] loaded metadata for tablet 9d67a811c1734c13b9119ed59e8ad1fb (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b])
I20250903 14:23:08.989181 1936 tablet_loader.cc:96] loaded metadata for tablet 9e61f697da744b0099dbd6939c1bc57c (table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b])
I20250903 14:23:08.989943 1936 tablet_loader.cc:96] loaded metadata for tablet b5afc9676cca4a9ba1841546f5d38d7a (table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9])
I20250903 14:23:08.990361 1936 tablet_loader.cc:96] loaded metadata for tablet c51713a4e7554285ba265825b8651efe (table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1])
I20250903 14:23:08.991001 1936 tablet_loader.cc:96] loaded metadata for tablet d1fcf42e749a463c8f47e275c0fe3439 (table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483])
I20250903 14:23:08.991693 1936 tablet_loader.cc:96] loaded metadata for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e])
I20250903 14:23:08.992600 1936 tablet_loader.cc:96] loaded metadata for tablet e6db03a8c88941278f0ab15599e4f975 (table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5])
I20250903 14:23:08.993333 1936 tablet_loader.cc:96] loaded metadata for tablet efadfb3a6ed5403e9d17f47891ac2206 (table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d])
I20250903 14:23:08.993783 1936 tablet_loader.cc:96] loaded metadata for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328])
I20250903 14:23:08.994226 1936 tablet_loader.cc:96] loaded metadata for tablet f9c4e2c45d19494f86c3c559d3d91202 (table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63])
I20250903 14:23:08.994958 1936 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:23:08.996943 1936 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:08.997037 1936 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:23:08.998783 1936 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:23:09.000063 1936 catalog_manager.cc:5966] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Loaded TSK: 0
I20250903 14:23:09.000743 1936 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:23:09.001859 1936 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 14:23:09.003194 1936 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250903 14:23:09.017071 757 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e78a7c3701f464080b7b887574a6b7a from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:09.017362 759 master_service.cc:948] ReplaceTablet: received request to replace tablet f9c4e2c45d19494f86c3c559d3d91202 from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:09.017593 760 master_service.cc:948] ReplaceTablet: received request to replace tablet b5afc9676cca4a9ba1841546f5d38d7a from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:09.018433 755 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_2e6fe8bc133e4eb68510d15bd758709a" } new_table_name: "default.Table_54d50ef1cebc4d34a9a9521fb7b13929" modify_external_catalogs: true
I20250903 14:23:09.018901 754 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_31acfe71a94349f187dc3fa60a6ecc89" } new_table_name: "default.Table_2c0dd73e42f04addad73b6089d755a89" modify_external_catalogs: true
I20250903 14:23:09.019110 758 master_service.cc:948] ReplaceTablet: received request to replace tablet d1fcf42e749a463c8f47e275c0fe3439 from {username='slave'} at 127.0.0.1:39614
I20250903 14:23:09.019340 753 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_8eadddc09609471d8b5295f2b97e9474" } new_table_name: "default.Table_dfb18ce4fe3a4f019997b07383cebffe" modify_external_catalogs: true
I20250903 14:23:09.019780 756 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_867ae72d30fd4ea5b2e85874f80942ad" } new_table_name: "default.Table_90d54f6435f34b629cd27486128824e1" modify_external_catalogs: true
I20250903 14:23:09.019797 752 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_6955211ac77d4f9cb6e5314ebdebf3ce" } modify_external_catalogs: true
I20250903 14:23:09.020183 752 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39614:
table { table_name: "default.Table_6955211ac77d4f9cb6e5314ebdebf3ce" } modify_external_catalogs: true
I20250903 14:23:09.022972 761 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39614:
name: "default.Table_1b9ce2c993734dcfa81dc61644f332bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 14:23:09.027596 746 consensus_peers.cc:489] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 -> Peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Couldn't send request to peer 2bbe08ab62bf4073825bf1b598ad2edb. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 14:23:09.032248 1879 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Refusing update from remote peer f4c58d929e53460883d05e0f14f20780: Log matching property violated. Preceding OpId in replica: term: 2 index: 80. Preceding OpId from leader: term: 4 index: 87. (index mismatch)
I20250903 14:23:09.033509 1935 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 79, Time since last communication: 0.000s
F20250903 14:23:09.035583 924 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 28, previous 28; dumping first 7 out of 10 received events: NotificationEvent(eventId=23, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_152f6fe6ff314070b2b909c120bf6a10, message={"server":"","servicePrincipal":"","db":"default","table":"table_152f6fe6ff314070b2b909c120bf6a10","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_152f6fe6ff314070b2b909c120bf6a10\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_152f6fe6ff314070b2b909c120bf6a10\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_152f6fe6ff314070b2b909c120bf6a10\",\"kudu.table_id\":\"4ad47717c2424f57a3235322430e8d3c\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_219a289c9b7a44af91e47feece52778a, message={"server":"","servicePrincipal":"","db":"default","table":"table_219a289c9b7a44af91e47feece52778a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_219a289c9b7a44af91e47feece52778a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_219a289c9b7a44af91e47feece52778a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_219a289c9b7a44af91e47feece52778a\",\"kudu.table_id\":\"03d2198ad57f4f729123cf099287a61d\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_a8c65bc030834131a6a9b071e7cae292, message={"server":"","servicePrincipal":"","db":"default","table":"table_a8c65bc030834131a6a9b071e7cae292","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a8c65bc030834131a6a9b071e7cae292\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_a8c65bc030834131a6a9b071e7cae292\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a8c65bc030834131a6a9b071e7cae292\",\"kudu.table_id\":\"ba26d86544084bfba8a647142ec79ed1\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_6fd4c0ea3f684a06b60c606e942d3961, message={"server":"","servicePrincipal":"","db":"default","table":"table_6fd4c0ea3f684a06b60c606e942d3961","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6fd4c0ea3f684a06b60c606e942d3961\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_6fd4c0ea3f684a06b60c606e942d3961\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6fd4c0ea3f684a06b60c606e942d3961\",\"kudu.table_id\":\"2cd7578b26ee4765836cf4c316fdcfb3\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_d5a7e44d7d1c4005aa2bc39c855d2418, message={"server":"","servicePrincipal":"","db":"default","table":"table_d5a7e44d7d1c4005aa2bc39c855d2418","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d5a7e44d7d1c4005aa2bc39c855d2418\",\"kudu.table_id\":\"d35e7f78eeb345df9ad80071d3b6ccb9\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_0ba72da4f2b84fb39612871a39e8d19f, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba72da4f2b84fb39612871a39e8d19f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba72da4f2b84fb39612871a39e8d19f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_0ba72da4f2b84fb39612871a39e8d19f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba72da4f2b84fb39612871a39e8d19f\",\"kudu.table_id\":\"8649e9b9d8ea44429fd251a29586363f\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_d63d2c9adbc5421f9a1c01f637efe55b, message={"server":"","servicePrincipal":"","db":"default","table":"table_d63d2c9adbc5421f9a1c01f637efe55b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d63d2c9adbc5421f9a1c01f637efe55b\",\"kudu.table_id\":\"6f5c11cb07914fb299328948ea5a0516\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f99506d8dcd google::LogMessage::Fail() at ??:0
@ 0x7f99506dcb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f99506d87cc google::LogMessage::Flush() at ??:0
@ 0x7f99506d8a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9966b1b7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250903 14:23:09.060978 1908 mvcc.cc:204] Tried to move back new op lower bound from 7196300855026663424 to 7196300851736686592. Current Snapshot: MvccSnapshot[applied={T|T < 7196300855026663424 or (T in {7196300855026663424})}]
I20250903 14:23:09.062361 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 4 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:09.062731 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9966b1a061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250903 14:23:09.063254 1941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:09.063642 1941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9966b21cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250903 14:23:09.071906 1941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:09.072170 1941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:09.073711 760 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Sending DeleteTablet for 3 replicas of tablet b5afc9676cca4a9ba1841546f5d38d7a
I20250903 14:23:09.074331 760 catalog_manager.cc:6652] ReplaceTablet: tablet b5afc9676cca4a9ba1841546f5d38d7a deleted and replaced by tablet 94e8a876084f485788dd3e93c58d5986
I20250903 14:23:09.076181 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet b5afc9676cca4a9ba1841546f5d38d7a with delete_type TABLET_DATA_DELETED (replaced by tablet 94e8a876084f485788dd3e93c58d5986) from {username='slave'} at 127.0.0.1:55526
I20250903 14:23:09.076606 1273 tablet_service.cc:1515] Processing DeleteTablet for tablet b5afc9676cca4a9ba1841546f5d38d7a with delete_type TABLET_DATA_DELETED (replaced by tablet 94e8a876084f485788dd3e93c58d5986) from {username='slave'} at 127.0.0.1:58704
@ 0x7f9966b21b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250903 14:23:09.077080 1136 tablet_service.cc:1515] Processing DeleteTablet for tablet b5afc9676cca4a9ba1841546f5d38d7a with delete_type TABLET_DATA_DELETED (replaced by tablet 94e8a876084f485788dd3e93c58d5986) from {username='slave'} at 127.0.0.1:42548
@ 0xa774dd std::function<>::operator()() at ??:?
I20250903 14:23:09.078780 1948 tablet_replica.cc:331] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: stopping tablet replica
I20250903 14:23:09.079078 1948 raft_consensus.cc:2241] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:09.079324 1948 raft_consensus.cc:2270] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:09.080125 1948 ts_tablet_manager.cc:1905] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f99534412b9 kudu::Thread::SuperviseThread() at ??:0
I20250903 14:23:09.083931 1949 tablet_replica.cc:331] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: stopping tablet replica
I20250903 14:23:09.084223 1949 raft_consensus.cc:2241] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 14:23:09.084489 1949 raft_consensus.cc:2270] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:09.085246 1949 ts_tablet_manager.cc:1905] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f99568656db start_thread at ??:0
I20250903 14:23:09.087883 1950 tablet_replica.cc:331] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: stopping tablet replica
@ 0x7f994f58971f clone at ??:0
I20250903 14:23:09.088075 759 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Sending DeleteTablet for 3 replicas of tablet f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:09.088150 1950 raft_consensus.cc:2241] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 LEADER]: Raft consensus shutting down.
I20250903 14:23:09.090288 1950 raft_consensus.cc:2270] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 14:23:09.090955 1950 ts_tablet_manager.cc:1905] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 14:23:09.093290 1948 ts_tablet_manager.cc:1918] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:09.093421 1948 log.cc:1199] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-0/wal/wals/b5afc9676cca4a9ba1841546f5d38d7a
I20250903 14:23:09.093842 1948 ts_tablet_manager.cc:1939] T b5afc9676cca4a9ba1841546f5d38d7a P 1adb02d7e9fc4548aa7fd31323ff8ecd: Deleting consensus metadata
I20250903 14:23:09.094014 1949 ts_tablet_manager.cc:1918] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:09.094208 1949 log.cc:1199] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-2/wal/wals/b5afc9676cca4a9ba1841546f5d38d7a
I20250903 14:23:09.094641 1949 ts_tablet_manager.cc:1939] T b5afc9676cca4a9ba1841546f5d38d7a P b03995cb49e94f108cf8caa7a502eafe: Deleting consensus metadata
I20250903 14:23:09.097615 1950 ts_tablet_manager.cc:1918] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 14:23:09.097728 1950 log.cc:1199] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: Deleting WAL directory at /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/ts-1/wal/wals/b5afc9676cca4a9ba1841546f5d38d7a
I20250903 14:23:09.098037 1950 ts_tablet_manager.cc:1939] T b5afc9676cca4a9ba1841546f5d38d7a P 5d4ed06db54c4b5199e190b837bc53fe: Deleting consensus metadata
I20250903 14:23:09.137683 1868 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:53357; Asking this server to re-register.
I20250903 14:23:09.139104 1067 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:09.141777 1868 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:09.220499 1204 heartbeater.cc:499] Master 127.0.104.62:37899 was elected leader, sending a full tablet report...
I20250903 14:23:09.223845 1868 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:36855; Asking this server to re-register.
I20250903 14:23:09.224963 1204 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:09.227231 1868 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:09.235867 1782 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:09.236119 1782 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:09.236810 1782 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:09.239926 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:23:09.244788 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:09.245000 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:09.245375 1782 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:23:09.331903 1234 connection.cc:537] server connection from 127.0.0.1:58704 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:09.331990 1354 connection.cc:537] client connection to 127.0.104.61:40437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:09.334385 1232 connection.cc:537] client connection to 127.0.104.61:40437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:09.335160 1093 connection.cc:537] client connection to 127.0.104.61:40437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:09.335983 956 connection.cc:537] client connection to 127.0.104.61:40437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:09.336220 955 connection.cc:537] server connection from 127.0.0.1:55526 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:09.337044 1093 connection.cc:537] server connection from 127.0.0.1:42548 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 14:23:09.349475 1868 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:42829; Asking this server to re-register.
I20250903 14:23:09.350329 1343 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:09.352331 1869 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:09.566660 1941 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:09.566883 1941 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:09.568836 1941 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:09.570022 1854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:09.571760 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:09.572052 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:09.572162 1854 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:09.572520 1941 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 14:23:10.214984 1971 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:10.215350 1971 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:10.216656 1971 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:10.221362 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:10.221602 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:10.221719 1854 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:10.222119 1971 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 14:23:10.948894 1496 raft_consensus.cc:491] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1adb02d7e9fc4548aa7fd31323ff8ecd)
I20250903 14:23:10.949163 1496 raft_consensus.cc:513] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:10.949823 1496 leader_election.cc:290] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:10.950532 1157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ec2abff20f1413fad17fd393d8fd1e9" candidate_uuid: "b03995cb49e94f108cf8caa7a502eafe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5d4ed06db54c4b5199e190b837bc53fe" is_pre_election: true
W20250903 14:23:10.951203 1234 leader_election.cc:343] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9
I20250903 14:23:10.951561 1234 leader_election.cc:304] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b03995cb49e94f108cf8caa7a502eafe; no voters: 1adb02d7e9fc4548aa7fd31323ff8ecd, 5d4ed06db54c4b5199e190b837bc53fe
I20250903 14:23:10.951833 1496 raft_consensus.cc:2747] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 14:23:11.164954 1985 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:11.165161 1985 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:11.165727 1985 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:11.167922 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:11.168128 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:11.168221 1854 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:11.168537 1985 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 14:23:11.479156 1981 raft_consensus.cc:491] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1adb02d7e9fc4548aa7fd31323ff8ecd)
I20250903 14:23:11.479357 1981 raft_consensus.cc:513] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:11.480276 1981 leader_election.cc:290] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:11.480845 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ec2abff20f1413fad17fd393d8fd1e9" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe" is_pre_election: true
I20250903 14:23:11.481189 1296 raft_consensus.cc:2466] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 1.
W20250903 14:23:11.481194 1095 leader_election.cc:343] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9
I20250903 14:23:11.481650 1095 leader_election.cc:304] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters: 1adb02d7e9fc4548aa7fd31323ff8ecd
I20250903 14:23:11.481986 1981 raft_consensus.cc:2802] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 14:23:11.482085 1981 raft_consensus.cc:491] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1adb02d7e9fc4548aa7fd31323ff8ecd)
I20250903 14:23:11.482162 1981 raft_consensus.cc:3058] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 1 FOLLOWER]: Advancing to term 2
I20250903 14:23:11.484076 1981 raft_consensus.cc:513] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:11.484591 1981 leader_election.cc:290] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 2 election: Requested vote from peers 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:11.485255 1296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ec2abff20f1413fad17fd393d8fd1e9" candidate_uuid: "5d4ed06db54c4b5199e190b837bc53fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b03995cb49e94f108cf8caa7a502eafe"
I20250903 14:23:11.485492 1296 raft_consensus.cc:3058] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 1 FOLLOWER]: Advancing to term 2
W20250903 14:23:11.485566 1095 leader_election.cc:343] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9
I20250903 14:23:11.487390 1296 raft_consensus.cc:2466] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4ed06db54c4b5199e190b837bc53fe in term 2.
I20250903 14:23:11.487957 1095 leader_election.cc:304] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe; no voters: 1adb02d7e9fc4548aa7fd31323ff8ecd
I20250903 14:23:11.488294 1981 raft_consensus.cc:2802] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 2 FOLLOWER]: Leader election won for term 2
I20250903 14:23:11.488688 1981 raft_consensus.cc:695] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [term 2 LEADER]: Becoming Leader. State: Replica: 5d4ed06db54c4b5199e190b837bc53fe, State: Running, Role: LEADER
I20250903 14:23:11.489197 1981 consensus_queue.cc:237] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:11.838889 1782 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:11.839080 1782 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:11.839711 1782 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:11.840646 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:11.840946 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:11.841068 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:11.841341 1782 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:12.028061 2000 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:12.028286 2000 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:12.028836 2000 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:12.031495 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:12.031704 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:12.031816 1854 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:12.032151 2000 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 14:23:12.168258 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--webserver_interface=127.0.104.61
--webserver_port=43811
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 14:23:12.267172 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 1: this message will repeat every 5th retry.
I20250903 14:23:12.289125 1296 raft_consensus.cc:1273] T 8ec2abff20f1413fad17fd393d8fd1e9 P b03995cb49e94f108cf8caa7a502eafe [term 2 FOLLOWER]: Refusing update from remote peer 5d4ed06db54c4b5199e190b837bc53fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250903 14:23:12.289793 1995 consensus_queue.cc:1035] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250903 14:23:12.326398 1782 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:12.326738 1782 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:12.327517 1782 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:12.328744 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:23:12.329443 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:12.329571 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:12.329859 1782 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:23:12.525135 2004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:12.525460 2004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:12.525592 2004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:12.535029 2004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:12.535132 2004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:12.535199 2004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:12.535274 2004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:12.545321 2004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=43811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:12.546954 2004 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:12.548789 2004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:12.557044 2015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:12.557158 2014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:12.557945 2017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:12.558282 2004 server_base.cc:1047] running on GCE node
I20250903 14:23:12.558998 2004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:12.560444 2004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:12.561657 2004 hybrid_clock.cc:648] HybridClock initialized: now 1756909392561617 us; error 37 us; skew 500 ppm
I20250903 14:23:12.561991 2004 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:12.565044 2004 webserver.cc:480] Webserver started at http://127.0.104.61:43811/ using document root <none> and password file <none>
I20250903 14:23:12.565754 2004 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:12.565857 2004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:12.571326 2004 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250903 14:23:12.574327 2023 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:12.575731 2004 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250903 14:23:12.575870 2004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
uuid: "f4c58d929e53460883d05e0f14f20780"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:23:12.576587 2004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:12.607375 2004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:12.608307 2004 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:12.608582 2004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:12.629855 2004 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:40437
I20250903 14:23:12.629899 2075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:40437 every 8 connection(s)
I20250903 14:23:12.631531 2004 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
I20250903 14:23:12.638058 2076 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:12.638584 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 2004
I20250903 14:23:12.642030 2076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap starting.
I20250903 14:23:12.660756 2076 log.cc:826] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:12.726925 2076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=88 ignored=0} inserts{seen=58 ignored=0} mutations{seen=69 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250903 14:23:12.727595 2076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap complete.
I20250903 14:23:12.733127 2076 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 4 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:12.736114 2076 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Initialized, Role: FOLLOWER
I20250903 14:23:12.736759 2076 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 4.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:12.737473 2082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:12.737720 2082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:12.738559 2076 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:23:12.740886 2087 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:12.741118 2087 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:12.741796 2087 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:12.746883 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:12.751909 2050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 5 candidate_status { last_received { term: 4 index: 90 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780" is_pre_election: true
I20250903 14:23:12.752607 2050 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e83636ce3ca47f3bfc9f84f6755eec2 in term 4.
I20250903 14:23:12.753394 1854 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb
I20250903 14:23:12.759611 2087 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250903 14:23:12.759784 2087 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:12.759900 2087 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 4 FOLLOWER]: Advancing to term 5
I20250903 14:23:12.763002 2087 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:12.764875 2087 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 election: Requested vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:12.767331 2100 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:12.767493 2100 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: loading cluster ID for follower catalog manager: success
I20250903 14:23:12.772156 2100 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: acquiring CA information for follower catalog manager: success
I20250903 14:23:12.776759 2050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" candidate_term: 5 candidate_status { last_received { term: 4 index: 90 } } ignore_live_leader: false dest_uuid: "f4c58d929e53460883d05e0f14f20780"
I20250903 14:23:12.777031 2050 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 4 FOLLOWER]: Advancing to term 5
W20250903 14:23:12.779672 1854 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:12.781358 2100 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:12.783665 2050 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e83636ce3ca47f3bfc9f84f6755eec2 in term 5.
I20250903 14:23:12.785715 1854 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb
I20250903 14:23:12.786125 2087 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 5 FOLLOWER]: Leader election won for term 5
I20250903 14:23:12.787890 2087 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 5 LEADER]: Becoming Leader. State: Replica: 2e83636ce3ca47f3bfc9f84f6755eec2, State: Running, Role: LEADER
I20250903 14:23:12.788519 2087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 4.90, Last appended by leader: 90, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:12.790560 2105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e83636ce3ca47f3bfc9f84f6755eec2. Latest consensus state: current_term: 5 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:12.790865 2105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:12.792835 2107 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:23:12.805459 2107 catalog_manager.cc:671] Loaded metadata for table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]
I20250903 14:23:12.806133 2107 catalog_manager.cc:671] Loaded metadata for table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483]
I20250903 14:23:12.806666 2107 catalog_manager.cc:671] Loaded metadata for table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]
I20250903 14:23:12.807152 2107 catalog_manager.cc:671] Loaded metadata for table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3]
I20250903 14:23:12.807611 2107 catalog_manager.cc:671] Loaded metadata for table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c]
I20250903 14:23:12.808133 2107 catalog_manager.cc:671] Loaded metadata for table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]
I20250903 14:23:12.808825 2107 catalog_manager.cc:671] Loaded metadata for table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1]
I20250903 14:23:12.809430 2107 catalog_manager.cc:671] Loaded metadata for table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]
I20250903 14:23:12.810276 2107 catalog_manager.cc:671] Loaded metadata for table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]
I20250903 14:23:12.810819 2107 catalog_manager.cc:671] Loaded metadata for table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]
I20250903 14:23:12.811278 2107 catalog_manager.cc:671] Loaded metadata for table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1]
I20250903 14:23:12.811836 2107 catalog_manager.cc:671] Loaded metadata for table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]
I20250903 14:23:12.812294 2107 catalog_manager.cc:671] Loaded metadata for table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]
I20250903 14:23:12.812800 2107 catalog_manager.cc:671] Loaded metadata for table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]
I20250903 14:23:12.813289 2107 catalog_manager.cc:671] Loaded metadata for table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]
I20250903 14:23:12.814078 1347 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:12.817315 2107 tablet_loader.cc:96] loaded metadata for tablet 29e447cc964646169fbe6ac64090b63e (table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2])
I20250903 14:23:12.817600 2039 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:58609; Asking this server to re-register.
I20250903 14:23:12.818158 2107 tablet_loader.cc:96] loaded metadata for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13])
I20250903 14:23:12.818638 1347 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:12.819243 2107 tablet_loader.cc:96] loaded metadata for tablet 72cce08a5fb14bf6ad95834dd56c3305 (table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c])
I20250903 14:23:12.819727 2107 tablet_loader.cc:96] loaded metadata for tablet 77c8726942e045ebac2610ab6b40e743 (table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1])
I20250903 14:23:12.821002 2107 tablet_loader.cc:96] loaded metadata for tablet 94e8a876084f485788dd3e93c58d5986 (table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9])
I20250903 14:23:12.821033 2039 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:12.821535 2107 tablet_loader.cc:96] loaded metadata for tablet 9a654351883e481496a2778b861fddfc (table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3])
I20250903 14:23:12.821995 2107 tablet_loader.cc:96] loaded metadata for tablet 9d67a811c1734c13b9119ed59e8ad1fb (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b])
I20250903 14:23:12.822451 2107 tablet_loader.cc:96] loaded metadata for tablet 9e61f697da744b0099dbd6939c1bc57c (table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b])
I20250903 14:23:12.823107 2107 tablet_loader.cc:96] loaded metadata for tablet b53d52eb9b4d468bac680dc8cffb597e (table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483])
I20250903 14:23:12.823808 2107 tablet_loader.cc:96] loaded metadata for tablet c51713a4e7554285ba265825b8651efe (table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1])
I20250903 14:23:12.824983 2107 tablet_loader.cc:96] loaded metadata for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e])
I20250903 14:23:12.825361 2107 tablet_loader.cc:96] loaded metadata for tablet e3927ba44aff4baf8d358f638f18d136 (table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63])
I20250903 14:23:12.826027 2107 tablet_loader.cc:96] loaded metadata for tablet e6db03a8c88941278f0ab15599e4f975 (table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5])
I20250903 14:23:12.826427 2107 tablet_loader.cc:96] loaded metadata for tablet efadfb3a6ed5403e9d17f47891ac2206 (table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d])
I20250903 14:23:12.826846 2107 tablet_loader.cc:96] loaded metadata for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328])
I20250903 14:23:12.827461 2107 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:23:12.823827 2004 master_runner.cc:186] Error getting master registration for 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111),
I20250903 14:23:12.829411 2107 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:12.829494 2107 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:23:12.829545 2004 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:12.831722 2107 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:23:12.833053 2107 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Loaded TSK: 0
I20250903 14:23:12.833421 1206 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:12.834262 2107 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:23:12.835703 2107 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 14:23:12.835862 2039 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:49991; Asking this server to re-register.
I20250903 14:23:12.836661 1206 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:12.837062 2107 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250903 14:23:12.838408 2039 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:12.848757 1869 master_service.cc:948] ReplaceTablet: received request to replace tablet 29e447cc964646169fbe6ac64090b63e from {username='slave'} at 127.0.0.1:47464
I20250903 14:23:12.850293 1860 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8eb8eda2d5fd408a87a960fe890af54c" } modify_external_catalogs: true
I20250903 14:23:12.850147 1862 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_663b9a72bf8543eb90c51bd282a2db4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:12.850610 1860 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8eb8eda2d5fd408a87a960fe890af54c" } modify_external_catalogs: true
I20250903 14:23:12.851033 1868 master_service.cc:948] ReplaceTablet: received request to replace tablet b5afc9676cca4a9ba1841546f5d38d7a from {username='slave'} at 127.0.0.1:47464
I20250903 14:23:12.851835 1868 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dd32253e3975467098b9075da837375c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:12.851711 1866 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d63d2c9adbc5421f9a1c01f637efe55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:12.852344 1864 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d7a178c8bc1546dd9464cf497775d66a" } modify_external_catalogs: true
I20250903 14:23:12.852490 1867 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb580089aaa843ff90baf8a5795d8f56" } new_table_name: "default.Table_37cdaa8349a94cf1abcf2b79b208eeee" modify_external_catalogs: true
I20250903 14:23:12.852608 1864 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d7a178c8bc1546dd9464cf497775d66a" } modify_external_catalogs: true
I20250903 14:23:12.853053 1865 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3368c82b8e6b4b149e8ad8d925c3d0e0" } modify_external_catalogs: true
I20250903 14:23:12.853125 1861 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0ba72da4f2b84fb39612871a39e8d19f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:12.853317 1865 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3368c82b8e6b4b149e8ad8d925c3d0e0" } modify_external_catalogs: true
I20250903 14:23:12.853188 1863 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e3decc796a014477b22f229bd0211387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250903 14:23:12.862677 1925 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 28, previous 28; dumping first 7 out of 10 received events: NotificationEvent(eventId=23, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_152f6fe6ff314070b2b909c120bf6a10, message={"server":"","servicePrincipal":"","db":"default","table":"table_152f6fe6ff314070b2b909c120bf6a10","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_152f6fe6ff314070b2b909c120bf6a10\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_152f6fe6ff314070b2b909c120bf6a10\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_152f6fe6ff314070b2b909c120bf6a10\",\"kudu.table_id\":\"4ad47717c2424f57a3235322430e8d3c\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_219a289c9b7a44af91e47feece52778a, message={"server":"","servicePrincipal":"","db":"default","table":"table_219a289c9b7a44af91e47feece52778a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_219a289c9b7a44af91e47feece52778a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_219a289c9b7a44af91e47feece52778a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_219a289c9b7a44af91e47feece52778a\",\"kudu.table_id\":\"03d2198ad57f4f729123cf099287a61d\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_a8c65bc030834131a6a9b071e7cae292, message={"server":"","servicePrincipal":"","db":"default","table":"table_a8c65bc030834131a6a9b071e7cae292","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a8c65bc030834131a6a9b071e7cae292\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_a8c65bc030834131a6a9b071e7cae292\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a8c65bc030834131a6a9b071e7cae292\",\"kudu.table_id\":\"ba26d86544084bfba8a647142ec79ed1\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_6fd4c0ea3f684a06b60c606e942d3961, message={"server":"","servicePrincipal":"","db":"default","table":"table_6fd4c0ea3f684a06b60c606e942d3961","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6fd4c0ea3f684a06b60c606e942d3961\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_6fd4c0ea3f684a06b60c606e942d3961\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6fd4c0ea3f684a06b60c606e942d3961\",\"kudu.table_id\":\"2cd7578b26ee4765836cf4c316fdcfb3\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_d5a7e44d7d1c4005aa2bc39c855d2418, message={"server":"","servicePrincipal":"","db":"default","table":"table_d5a7e44d7d1c4005aa2bc39c855d2418","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d5a7e44d7d1c4005aa2bc39c855d2418\",\"kudu.table_id\":\"d35e7f78eeb345df9ad80071d3b6ccb9\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_0ba72da4f2b84fb39612871a39e8d19f, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba72da4f2b84fb39612871a39e8d19f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba72da4f2b84fb39612871a39e8d19f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_0ba72da4f2b84fb39612871a39e8d19f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba72da4f2b84fb39612871a39e8d19f\",\"kudu.table_id\":\"8649e9b9d8ea44429fd251a29586363f\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_d63d2c9adbc5421f9a1c01f637efe55b, message={"server":"","servicePrincipal":"","db":"default","table":"table_d63d2c9adbc5421f9a1c01f637efe55b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d63d2c9adbc5421f9a1c01f637efe55b\",\"kudu.table_id\":\"6f5c11cb07914fb299328948ea5a0516\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
W20250903 14:23:12.864154 1854 consensus_peers.cc:489] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 -> Peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Couldn't send request to peer 2bbe08ab62bf4073825bf1b598ad2edb. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 14:23:12.864615 2050 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Refusing update from remote peer 2e83636ce3ca47f3bfc9f84f6755eec2: Log matching property violated. Preceding OpId in replica: term: 4 index: 90. Preceding OpId from leader: term: 5 index: 92. (index mismatch)
@ 0x7fea6ca0edcd google::LogMessage::Fail() at ??:0
I20250903 14:23:12.865398 2105 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 88, Time since last communication: 0.000s
@ 0x7fea6ca12b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fea6ca0e7cc google::LogMessage::Flush() at ??:0
@ 0x7fea6ca0ea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fea82e517b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fea82e50061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fea82e57cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250903 14:23:12.890859 2082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e83636ce3ca47f3bfc9f84f6755eec2. Latest consensus state: current_term: 5 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:12.891177 2082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:12.896657 2082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:12.896883 2082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fea82e57b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250903 14:23:12.912307 2087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:12.912614 2087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: LEADER
@ 0xa774dd std::function<>::operator()() at ??:?
I20250903 14:23:12.913530 1869 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Sending DeleteTablet for 3 replicas of tablet 29e447cc964646169fbe6ac64090b63e
@ 0x7fea6f7772b9 kudu::Thread::SuperviseThread() at ??:0
I20250903 14:23:12.919067 1869 catalog_manager.cc:6652] ReplaceTablet: tablet 29e447cc964646169fbe6ac64090b63e deleted and replaced by tablet ec9f095964664809a78daebd0f25dad7
@ 0x7fea72b9b6db start_thread at ??:0
@ 0x7fea6b8bf71f clone at ??:0
I20250903 14:23:13.042163 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:13.044878 2039 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:59031; Asking this server to re-register.
I20250903 14:23:13.045671 1069 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:13.047715 2039 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
W20250903 14:23:13.121676 2128 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:12.920379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.104.2:46863)
0903 14:23:12.930853 (+ 10474us) server_negotiation.cc:207] Beginning negotiation
0903 14:23:12.930860 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 14:23:13.121353 (+190493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56684: BlockingRecv error: recv got EOF from 127.0.0.1:56684 (error 108)
Metrics: {"server-negotiator.queue_time_us":10331,"thread_start_us":11527,"threads_started":1}
W20250903 14:23:13.121723 2125 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:12.918177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.104.3:43373)
0903 14:23:12.935821 (+ 17644us) server_negotiation.cc:207] Beginning negotiation
0903 14:23:12.935829 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 14:23:12.935853 (+ 24us) server_negotiation.cc:408] Connection header received
0903 14:23:12.935935 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:23:12.935943 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:23:12.936041 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:23:12.936199 (+ 158us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:23:13.121407 (+185208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35118: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":17488,"thread_start_us":17391,"threads_started":1}
W20250903 14:23:13.123260 2126 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 14:23:12.918566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.0.104.1:39849)
0903 14:23:12.931233 (+ 12667us) server_negotiation.cc:207] Beginning negotiation
0903 14:23:12.931242 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 14:23:12.931266 (+ 24us) server_negotiation.cc:408] Connection header received
0903 14:23:12.931354 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 14:23:12.931360 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 14:23:12.931479 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 14:23:12.931602 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 14:23:13.122241 (+190639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33002: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":12504,"thread_start_us":13235,"threads_started":1}
W20250903 14:23:13.123477 1356 connection.cc:537] client connection to 127.0.104.62:37899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:13.123287 958 connection.cc:537] client connection to 127.0.104.62:37899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:13.124136 1356 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:37899)
W20250903 14:23:13.124645 1356 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:37899)
W20250903 14:23:13.125087 1356 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:37899)
W20250903 14:23:13.125285 1356 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:37899)
W20250903 14:23:13.125991 1095 connection.cc:537] client connection to 127.0.104.62:37899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 14:23:13.396728 2082 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2e83636ce3ca47f3bfc9f84f6755eec2)
I20250903 14:23:13.396960 2082 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:13.398758 2082 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:13.399590 2027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:13.400965 2027 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:13.401211 2025 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:13.401311 2025 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:13.401674 2082 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:14.107087 2146 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2e83636ce3ca47f3bfc9f84f6755eec2)
I20250903 14:23:14.107280 2146 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:14.107973 2146 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:14.110072 2025 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:14.110244 2027 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:14.110370 2027 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:14.110651 2146 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:14.947361 2155 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 2e83636ce3ca47f3bfc9f84f6755eec2)
I20250903 14:23:14.947642 2155 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:14.948298 2155 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:14.950652 2025 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:14.950891 2027 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:14.951027 2027 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:14.951360 2155 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:15.185726 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu pid 2004
I20250903 14:23:15.192241 2071 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 14:23:15.303512 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu with pid 2004
I20250903 14:23:16.177716 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--webserver_interface=127.0.104.61
--webserver_port=43811
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 14:23:16.514808 2171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:16.515250 2171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:16.515381 2171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:16.524277 2171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:16.524400 2171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:16.524505 2171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:16.524576 2171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:16.534610 2171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=43811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:16.536326 2171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:16.538115 2171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:16.546918 2179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:16.547010 2178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:16.548081 2181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:16.548722 2171 server_base.cc:1047] running on GCE node
I20250903 14:23:16.549643 2171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:16.551469 2171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:16.552719 2171 hybrid_clock.cc:648] HybridClock initialized: now 1756909396552654 us; error 46 us; skew 500 ppm
I20250903 14:23:16.553098 2171 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:16.555330 2171 webserver.cc:480] Webserver started at http://127.0.104.61:43811/ using document root <none> and password file <none>
I20250903 14:23:16.556164 2171 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:16.556254 2171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:16.561592 2171 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 14:23:16.564615 2187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:16.565984 2171 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:23:16.566138 2171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
uuid: "f4c58d929e53460883d05e0f14f20780"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:23:16.566829 2171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:16.585740 2171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:16.586649 2171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:16.586941 2171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:16.606890 2171 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:40437
I20250903 14:23:16.606922 2238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:40437 every 8 connection(s)
I20250903 14:23:16.608649 2171 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
I20250903 14:23:16.615291 2239 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:16.616859 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 2171
I20250903 14:23:16.619475 2239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap starting.
I20250903 14:23:16.641191 2239 log.cc:826] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:16.706607 2239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=61 ignored=0} mutations{seen=72 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:23:16.707227 2239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap complete.
I20250903 14:23:16.712546 2239 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:16.713227 2239 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Initialized, Role: FOLLOWER
I20250903 14:23:16.713855 2239 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 5.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:16.714586 2243 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:16.714854 2243 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:16.715741 2239 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:23:16.725549 2171 master_runner.cc:186] Error getting master registration for 127.0.104.62:37899: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111),
I20250903 14:23:16.730096 2171 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:16.737246 2257 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:16.737355 2257 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: loading cluster ID for follower catalog manager: success
I20250903 14:23:16.742206 2257 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: acquiring CA information for follower catalog manager: success
I20250903 14:23:16.746130 2257 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:16.843365 1347 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:16.846499 2204 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:54355; Asking this server to re-register.
I20250903 14:23:16.847213 1347 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:16.849186 2204 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:16.941319 1206 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:16.943428 2204 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:51703; Asking this server to re-register.
I20250903 14:23:16.944033 1206 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:16.945483 2204 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:17.070211 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:17.071736 2204 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:58965; Asking this server to re-register.
I20250903 14:23:17.072343 1069 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:17.073832 2204 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:17.312158 2264 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:17.312419 2264 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:17.314173 2264 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:17.315169 2191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:17.316426 2189 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:17.316635 2191 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:17.316784 2191 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:17.317098 2264 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:17.449144 2056 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:17.449363 2056 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:17.450018 2056 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:17.450927 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:23:17.451113 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:17.451244 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:17.451525 2056 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:17.564280 2056 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:17.564481 2056 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:17.565135 2056 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:17.565932 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:17.566185 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:17.566275 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:17.566507 2056 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:18.052100 2273 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:18.052330 2273 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:18.053779 2273 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:18.055405 2189 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:18.055797 2191 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:18.055939 2191 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:18.056244 2273 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250903 14:23:18.517479 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 6: this message will repeat every 5th retry.
I20250903 14:23:18.826682 2283 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:18.826886 2283 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:18.827584 2283 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:18.829594 2189 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:18.829773 2191 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:18.829916 2191 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:18.830155 2283 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:19.206076 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu pid 2171
I20250903 14:23:19.218503 2234 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 14:23:19.328594 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu with pid 2171
I20250903 14:23:19.953465 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--webserver_interface=127.0.104.61
--webserver_port=43811
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 14:23:20.293000 2297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:20.293331 2297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:20.293419 2297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:20.301689 2297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:20.301793 2297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:20.301848 2297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:20.301895 2297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:20.311933 2297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=43811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:20.313546 2297 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:20.315301 2297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:20.323899 2302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:20.324249 2303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:20.324342 2305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:20.325608 2297 server_base.cc:1047] running on GCE node
I20250903 14:23:20.326390 2297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:20.327991 2297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:20.329206 2297 hybrid_clock.cc:648] HybridClock initialized: now 1756909400329160 us; error 40 us; skew 500 ppm
I20250903 14:23:20.329559 2297 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:20.331990 2297 webserver.cc:480] Webserver started at http://127.0.104.61:43811/ using document root <none> and password file <none>
I20250903 14:23:20.332769 2297 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:20.332861 2297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:20.338496 2297 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 14:23:20.341472 2311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:20.342804 2297 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 14:23:20.342988 2297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
uuid: "f4c58d929e53460883d05e0f14f20780"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:23:20.343708 2297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:20.365006 2297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:20.365887 2297 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:20.366282 2297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:20.387310 2297 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:40437
I20250903 14:23:20.387373 2364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:40437 every 8 connection(s)
I20250903 14:23:20.389112 2297 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
I20250903 14:23:20.392802 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 2297
I20250903 14:23:20.396816 2365 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:20.400315 2365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap starting.
I20250903 14:23:20.417554 2365 log.cc:826] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:20.490357 2365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=61 ignored=0} mutations{seen=72 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:23:20.491194 2365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap complete.
I20250903 14:23:20.497329 2365 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:20.498039 2365 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Initialized, Role: FOLLOWER
I20250903 14:23:20.498625 2365 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 5.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:20.499397 2370 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:20.499696 2370 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:20.500494 2365 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:23:20.515120 2384 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:20.515340 2384 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: loading cluster ID for follower catalog manager: success
I20250903 14:23:20.511368 2297 master_runner.cc:186] Error getting master registration for 127.0.104.62:37899: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111),
I20250903 14:23:20.517599 2297 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:20.520690 2384 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: acquiring CA information for follower catalog manager: success
I20250903 14:23:20.524331 2384 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:20.596335 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:20.596886 1206 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:20.599661 2329 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:56751; Asking this server to re-register.
I20250903 14:23:20.599684 2330 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:38997; Asking this server to re-register.
I20250903 14:23:20.600452 1069 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:20.600641 1206 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:20.602632 2329 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:20.603853 2330 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:20.869856 1347 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:20.871516 2330 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:46539; Asking this server to re-register.
I20250903 14:23:20.872157 1347 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:20.873651 2330 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:20.907779 2370 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:20.908084 2370 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:20.910218 2370 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:20.910857 2315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:20.912374 2315 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:20.912618 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:20.912731 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:20.913084 2370 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:21.530375 2397 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:21.530575 2397 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:21.531675 2397 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:21.534127 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:21.534399 2315 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:21.534498 2315 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:21.534737 2397 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:21.944551 2369 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:21.944743 2369 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:21.945560 2369 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:21.946586 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:21.946780 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:21.946883 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:21.947152 2369 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:23:22.113607 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 6: this message will repeat every 5th retry.
I20250903 14:23:22.423153 2405 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:22.423339 2405 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:22.424083 2405 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:22.426234 2315 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:22.426419 2313 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:22.426539 2313 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:22.426746 2405 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:22.706205 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu pid 2297
I20250903 14:23:22.713078 2360 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 14:23:22.819736 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu with pid 2297
W20250903 14:23:22.841728 1354 connection.cc:537] client connection to 127.0.104.61:40437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 14:23:23.652865 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--webserver_interface=127.0.104.61
--webserver_port=43811
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 14:23:23.990764 2415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:23.991101 2415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:23.991191 2415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:23.999701 2415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:23.999848 2415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:23.999936 2415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:24.000005 2415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:24.010300 2415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=43811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:24.012027 2415 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:24.013770 2415 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:24.021986 2425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:24.022045 2426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:24.022641 2428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:24.023036 2415 server_base.cc:1047] running on GCE node
I20250903 14:23:24.023874 2415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:24.025322 2415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:24.026569 2415 hybrid_clock.cc:648] HybridClock initialized: now 1756909404026520 us; error 34 us; skew 500 ppm
I20250903 14:23:24.026948 2415 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:24.029398 2415 webserver.cc:480] Webserver started at http://127.0.104.61:43811/ using document root <none> and password file <none>
I20250903 14:23:24.030275 2415 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:24.030390 2415 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:24.036057 2415 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 14:23:24.039245 2434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:24.040793 2415 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:23:24.041059 2415 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
uuid: "f4c58d929e53460883d05e0f14f20780"
format_stamp: "Formatted at 2025-09-03 14:22:59 on dist-test-slave-rxtz"
I20250903 14:23:24.041822 2415 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:24.057860 2415 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:24.058724 2415 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:24.059023 2415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:24.079272 2415 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:40437
I20250903 14:23:24.079295 2485 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:40437 every 8 connection(s)
I20250903 14:23:24.080992 2415 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-1/data/info.pb
I20250903 14:23:24.081699 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 2415
I20250903 14:23:24.089674 2486 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:24.093961 2486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap starting.
I20250903 14:23:24.110564 2486 log.cc:826] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:24.122202 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:24.180580 2486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=61 ignored=0} mutations{seen=72 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:23:24.181293 2486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Bootstrap complete.
I20250903 14:23:24.186787 2486 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:24.187511 2486 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Initialized, Role: FOLLOWER
I20250903 14:23:24.188133 2486 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 5.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:24.188938 2490 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:24.189208 2490 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:24.190181 2486 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:23:24.202769 2504 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:24.202951 2504 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: loading cluster ID for follower catalog manager: success
I20250903 14:23:24.201138 2415 master_runner.cc:186] Error getting master registration for 127.0.104.62:37899: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111),
I20250903 14:23:24.207082 2415 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:24.208451 2504 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: acquiring CA information for follower catalog manager: success
I20250903 14:23:24.212162 2504 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:24.392483 1347 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:24.394801 2451 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:39411; Asking this server to re-register.
I20250903 14:23:24.395505 1347 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:24.397382 2451 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:24.416687 1206 heartbeater.cc:344] Connected to a master server at 127.0.104.61:40437
I20250903 14:23:24.418728 2451 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:34033; Asking this server to re-register.
I20250903 14:23:24.419334 1206 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:24.420909 2451 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:24.627617 2451 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:42303; Asking this server to re-register.
I20250903 14:23:24.628280 1069 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:24.629737 2451 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:24.679569 2490 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:24.679831 2490 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:24.681895 2490 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:24.682384 2438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:24.683758 2438 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
W20250903 14:23:24.684005 2436 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:24.684124 2436 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:24.684525 2490 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:25.097123 2369 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:25.097312 2369 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:25.097926 2369 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:25.098830 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:25.099061 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:25.099161 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:25.099401 2369 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:25.368435 2515 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:25.368618 2515 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:25.369221 2515 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:25.372241 2436 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:25.372471 2438 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:25.372576 2438 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:25.372839 2515 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:26.422523 2519 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:26.422742 2519 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:26.423473 2519 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:26.424372 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:23:26.424580 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:26.424691 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:26.424949 2519 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:26.490468 2525 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:26.490741 2525 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:26.491545 2525 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:26.493695 2436 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:23:26.493861 2438 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.104.62:37899: connect: Connection refused (error 111)
I20250903 14:23:26.493996 2438 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, 2e83636ce3ca47f3bfc9f84f6755eec2
I20250903 14:23:26.494275 2525 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 14:23:27.421329 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--webserver_interface=127.0.104.62
--webserver_port=33279
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 14:23:27.755195 2534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:27.755578 2534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:27.755695 2534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:27.764009 2534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:27.764110 2534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:27.764187 2534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:27.764256 2534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:27.774564 2534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=33279
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:27.776342 2534 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:27.778162 2534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:27.786597 2544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:27.786597 2545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:27.787870 2547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:27.788615 2534 server_base.cc:1047] running on GCE node
I20250903 14:23:27.789502 2534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:27.791328 2534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:27.792531 2534 hybrid_clock.cc:648] HybridClock initialized: now 1756909407792481 us; error 42 us; skew 500 ppm
I20250903 14:23:27.792889 2534 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:27.795131 2534 webserver.cc:480] Webserver started at http://127.0.104.62:33279/ using document root <none> and password file <none>
I20250903 14:23:27.795917 2534 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:27.796025 2534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:27.801317 2534 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 14:23:27.804159 2553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:27.805557 2534 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 14:23:27.805716 2534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
format_stamp: "Formatted at 2025-09-03 14:22:58 on dist-test-slave-rxtz"
I20250903 14:23:27.806417 2534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:27.842523 2534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:27.843499 2534 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:27.843772 2534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:27.864213 2534 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37899
I20250903 14:23:27.864236 2604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37899 every 8 connection(s)
I20250903 14:23:27.865886 2534 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
I20250903 14:23:27.870419 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 2534
I20250903 14:23:27.873039 2605 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:27.876735 2605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap starting.
I20250903 14:23:27.895087 2605 log.cc:826] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Log is configured to *not* fsync() on all Append() calls
I20250903 14:23:27.964004 2605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=61 ignored=0} mutations{seen=72 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:23:27.964756 2605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap complete.
I20250903 14:23:27.970602 2605 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:27.971277 2605 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e83636ce3ca47f3bfc9f84f6755eec2, State: Initialized, Role: FOLLOWER
I20250903 14:23:27.971976 2605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 5.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:27.972781 2609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:27.973146 2609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:27.973920 2605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:23:27.997285 2623 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:27.997495 2623 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: loading cluster ID for follower catalog manager: success
I20250903 14:23:28.002466 2623 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: acquiring CA information for follower catalog manager: success
I20250903 14:23:28.006078 2623 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:28.012827 2534 master_runner.cc:186] Error getting master registration for 127.0.104.60:43143: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111),
I20250903 14:23:28.016443 2534 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:28.057322 2628 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:28.057567 2628 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:28.058307 2628 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:28.062001 2436 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:28.067206 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 6 candidate_status { last_received { term: 5 index: 92 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" is_pre_election: true
I20250903 14:23:28.067845 2580 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 5.
I20250903 14:23:28.068624 2438 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb
I20250903 14:23:28.068939 2628 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250903 14:23:28.069080 2628 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 14:23:28.069195 2628 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 5 FOLLOWER]: Advancing to term 6
I20250903 14:23:28.072741 2628 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:28.073352 2628 leader_election.cc:290] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 election: Requested vote from peers 2e83636ce3ca47f3bfc9f84f6755eec2 (127.0.104.62:37899), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
I20250903 14:23:28.073783 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4c58d929e53460883d05e0f14f20780" candidate_term: 6 candidate_status { last_received { term: 5 index: 92 } } ignore_live_leader: false dest_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
I20250903 14:23:28.074002 2580 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 5 FOLLOWER]: Advancing to term 6
W20250903 14:23:28.074784 2436 leader_election.cc:336] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:28.078464 2580 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c58d929e53460883d05e0f14f20780 in term 6.
I20250903 14:23:28.078882 2438 leader_election.cc:304] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2, f4c58d929e53460883d05e0f14f20780; no voters: 2bbe08ab62bf4073825bf1b598ad2edb
I20250903 14:23:28.079244 2628 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 6 FOLLOWER]: Leader election won for term 6
I20250903 14:23:28.079977 2628 raft_consensus.cc:695] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [term 6 LEADER]: Becoming Leader. State: Replica: f4c58d929e53460883d05e0f14f20780, State: Running, Role: LEADER
I20250903 14:23:28.080551 2628 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 5.92, Last appended by leader: 92, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:28.083251 2632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 6 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:28.083586 2632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [sys.catalog]: This master's current role is: LEADER
I20250903 14:23:28.088095 2634 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 14:23:28.091634 2634 catalog_manager.cc:671] Loaded metadata for table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d]
I20250903 14:23:28.092172 2634 catalog_manager.cc:671] Loaded metadata for table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483]
I20250903 14:23:28.092579 2634 catalog_manager.cc:671] Loaded metadata for table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63]
I20250903 14:23:28.093111 2634 catalog_manager.cc:671] Loaded metadata for table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3]
I20250903 14:23:28.093932 2634 catalog_manager.cc:671] Loaded metadata for table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c]
I20250903 14:23:28.094319 2634 catalog_manager.cc:671] Loaded metadata for table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5]
I20250903 14:23:28.094791 2634 catalog_manager.cc:671] Loaded metadata for table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1]
I20250903 14:23:28.095125 2634 catalog_manager.cc:671] Loaded metadata for table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b]
I20250903 14:23:28.095716 2634 catalog_manager.cc:671] Loaded metadata for table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13]
I20250903 14:23:28.096155 2634 catalog_manager.cc:671] Loaded metadata for table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2]
I20250903 14:23:28.096580 2634 catalog_manager.cc:671] Loaded metadata for table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1]
I20250903 14:23:28.097007 2634 catalog_manager.cc:671] Loaded metadata for table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e]
I20250903 14:23:28.097339 2634 catalog_manager.cc:671] Loaded metadata for table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b]
I20250903 14:23:28.097692 2634 catalog_manager.cc:671] Loaded metadata for table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328]
I20250903 14:23:28.098011 2634 catalog_manager.cc:671] Loaded metadata for table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9]
I20250903 14:23:28.100837 2634 tablet_loader.cc:96] loaded metadata for tablet 3a86369c9b3b4321abe742146a37623b (table default.table_bd0820d81cd1476c9b6474c12f0053f6 [id=8590620c66374843adbe167d46c1de13])
I20250903 14:23:28.101413 2634 tablet_loader.cc:96] loaded metadata for tablet 72cce08a5fb14bf6ad95834dd56c3305 (table default.table_152f6fe6ff314070b2b909c120bf6a10 [id=4ad47717c2424f57a3235322430e8d3c])
I20250903 14:23:28.101706 2634 tablet_loader.cc:96] loaded metadata for tablet 77c8726942e045ebac2610ab6b40e743 (table default.table_a8c65bc030834131a6a9b071e7cae292 [id=ba26d86544084bfba8a647142ec79ed1])
I20250903 14:23:28.102387 2634 tablet_loader.cc:96] loaded metadata for tablet 94e8a876084f485788dd3e93c58d5986 (table default.table_d5a7e44d7d1c4005aa2bc39c855d2418 [id=d35e7f78eeb345df9ad80071d3b6ccb9])
I20250903 14:23:28.102646 2634 tablet_loader.cc:96] loaded metadata for tablet 9a654351883e481496a2778b861fddfc (table default.table_6fd4c0ea3f684a06b60c606e942d3961 [id=2cd7578b26ee4765836cf4c316fdcfb3])
I20250903 14:23:28.102926 2634 tablet_loader.cc:96] loaded metadata for tablet 9d67a811c1734c13b9119ed59e8ad1fb (table default.table_6955211ac77d4f9cb6e5314ebdebf3ce [id=c8c7b6572d314c76ad67c3cab54e9b2b])
I20250903 14:23:28.103236 2634 tablet_loader.cc:96] loaded metadata for tablet 9e61f697da744b0099dbd6939c1bc57c (table default.table_8eb8eda2d5fd408a87a960fe890af54c [id=821eb02a05ea4da5a364c5e62037f88b])
I20250903 14:23:28.103683 2634 tablet_loader.cc:96] loaded metadata for tablet b53d52eb9b4d468bac680dc8cffb597e (table default.table_3368c82b8e6b4b149e8ad8d925c3d0e0 [id=2224cdb559fe45599b2464369cd2b483])
I20250903 14:23:28.104140 2634 tablet_loader.cc:96] loaded metadata for tablet c51713a4e7554285ba265825b8651efe (table default.table_2e6fe8bc133e4eb68510d15bd758709a [id=69ed0433e40249699066ac4a0b3825a1])
I20250903 14:23:28.104822 2634 tablet_loader.cc:96] loaded metadata for tablet dfc22043626840419ad42d3a55eb3849 (table default.table_31acfe71a94349f187dc3fa60a6ecc89 [id=c3c0e03ececf43e1a7c9c2c40acf188e])
I20250903 14:23:28.105082 2634 tablet_loader.cc:96] loaded metadata for tablet e3927ba44aff4baf8d358f638f18d136 (table default.table_8eadddc09609471d8b5295f2b97e9474 [id=23986959b59c41b190cf66a723f57e63])
I20250903 14:23:28.105522 2634 tablet_loader.cc:96] loaded metadata for tablet e6db03a8c88941278f0ab15599e4f975 (table default.table_867ae72d30fd4ea5b2e85874f80942ad [id=548535eb537048f489876c6c19d999f5])
I20250903 14:23:28.105760 2634 tablet_loader.cc:96] loaded metadata for tablet ec9f095964664809a78daebd0f25dad7 (table default.table_eb580089aaa843ff90baf8a5795d8f56 [id=9c7678e144ce4862b7e1db4771a0ada2])
I20250903 14:23:28.106030 2634 tablet_loader.cc:96] loaded metadata for tablet efadfb3a6ed5403e9d17f47891ac2206 (table default.table_219a289c9b7a44af91e47feece52778a [id=03d2198ad57f4f729123cf099287a61d])
I20250903 14:23:28.106303 2634 tablet_loader.cc:96] loaded metadata for tablet f2f7427e774b42b1bc3de1e70792f7c0 (table default.table_d7a178c8bc1546dd9464cf497775d66a [id=d237316ba9fd4366a139c31dd9524328])
I20250903 14:23:28.106732 2634 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 14:23:28.108369 2634 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:28.108467 2634 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 14:23:28.110838 2634 catalog_manager.cc:1506] Loading token signing keys...
I20250903 14:23:28.112536 2634 catalog_manager.cc:5966] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780: Loaded TSK: 0
I20250903 14:23:28.113623 2634 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 14:23:28.115356 2634 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 14:23:28.116590 2634 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250903 14:23:28.119468 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:28.123281 2570 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:33309; Asking this server to re-register.
I20250903 14:23:28.124114 1204 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:28.126058 2570 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:28.142635 2451 master_service.cc:948] ReplaceTablet: received request to replace tablet f2f7427e774b42b1bc3de1e70792f7c0 from {username='slave'} at 127.0.0.1:59568
I20250903 14:23:28.142728 2450 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e78a7c3701f464080b7b887574a6b7a from {username='slave'} at 127.0.0.1:59568
I20250903 14:23:28.142894 2449 master_service.cc:948] ReplaceTablet: received request to replace tablet f9c4e2c45d19494f86c3c559d3d91202 from {username='slave'} at 127.0.0.1:59568
I20250903 14:23:28.143034 2448 master_service.cc:948] ReplaceTablet: received request to replace tablet d1fcf42e749a463c8f47e275c0fe3439 from {username='slave'} at 127.0.0.1:59568
I20250903 14:23:28.144176 2443 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59568:
name: "default.Table_1b9ce2c993734dcfa81dc61644f332bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:28.146168 2442 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_6955211ac77d4f9cb6e5314ebdebf3ce" } modify_external_catalogs: true
I20250903 14:23:28.146477 2442 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_6955211ac77d4f9cb6e5314ebdebf3ce" } modify_external_catalogs: true
I20250903 14:23:28.147680 2448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_3368c82b8e6b4b149e8ad8d925c3d0e0" } modify_external_catalogs: true
I20250903 14:23:28.147910 2448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_3368c82b8e6b4b149e8ad8d925c3d0e0" } modify_external_catalogs: true
I20250903 14:23:28.148473 2445 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_2e6fe8bc133e4eb68510d15bd758709a" } new_table_name: "default.Table_54d50ef1cebc4d34a9a9521fb7b13929" modify_external_catalogs: true
I20250903 14:23:28.149112 2447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_8eadddc09609471d8b5295f2b97e9474" } new_table_name: "default.Table_dfb18ce4fe3a4f019997b07383cebffe" modify_external_catalogs: true
I20250903 14:23:28.149327 2449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59568:
name: "default.Table_d63d2c9adbc5421f9a1c01f637efe55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 14:23:28.149556 2450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_d7a178c8bc1546dd9464cf497775d66a" } modify_external_catalogs: true
I20250903 14:23:28.149770 2450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_d7a178c8bc1546dd9464cf497775d66a" } modify_external_catalogs: true
F20250903 14:23:28.151121 2503 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 28, previous 28; dumping first 7 out of 10 received events: NotificationEvent(eventId=23, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_152f6fe6ff314070b2b909c120bf6a10, message={"server":"","servicePrincipal":"","db":"default","table":"table_152f6fe6ff314070b2b909c120bf6a10","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_152f6fe6ff314070b2b909c120bf6a10\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_152f6fe6ff314070b2b909c120bf6a10\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_152f6fe6ff314070b2b909c120bf6a10\",\"kudu.table_id\":\"4ad47717c2424f57a3235322430e8d3c\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_219a289c9b7a44af91e47feece52778a, message={"server":"","servicePrincipal":"","db":"default","table":"table_219a289c9b7a44af91e47feece52778a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_219a289c9b7a44af91e47feece52778a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_219a289c9b7a44af91e47feece52778a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_219a289c9b7a44af91e47feece52778a\",\"kudu.table_id\":\"03d2198ad57f4f729123cf099287a61d\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_a8c65bc030834131a6a9b071e7cae292, message={"server":"","servicePrincipal":"","db":"default","table":"table_a8c65bc030834131a6a9b071e7cae292","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a8c65bc030834131a6a9b071e7cae292\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_a8c65bc030834131a6a9b071e7cae292\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a8c65bc030834131a6a9b071e7cae292\",\"kudu.table_id\":\"ba26d86544084bfba8a647142ec79ed1\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_6fd4c0ea3f684a06b60c606e942d3961, message={"server":"","servicePrincipal":"","db":"default","table":"table_6fd4c0ea3f684a06b60c606e942d3961","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_6fd4c0ea3f684a06b60c606e942d3961\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_6fd4c0ea3f684a06b60c606e942d3961\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6fd4c0ea3f684a06b60c606e942d3961\",\"kudu.table_id\":\"2cd7578b26ee4765836cf4c316fdcfb3\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756909386, eventType=DROP_TABLE, dbName=default, tableName=table_d5a7e44d7d1c4005aa2bc39c855d2418, message={"server":"","servicePrincipal":"","db":"default","table":"table_d5a7e44d7d1c4005aa2bc39c855d2418","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909383},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d5a7e44d7d1c4005aa2bc39c855d2418\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d5a7e44d7d1c4005aa2bc39c855d2418\",\"kudu.table_id\":\"d35e7f78eeb345df9ad80071d3b6ccb9\",\"transient_lastDdlTime\":\"1756909383\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_0ba72da4f2b84fb39612871a39e8d19f, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba72da4f2b84fb39612871a39e8d19f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba72da4f2b84fb39612871a39e8d19f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_0ba72da4f2b84fb39612871a39e8d19f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba72da4f2b84fb39612871a39e8d19f\",\"kudu.table_id\":\"8649e9b9d8ea44429fd251a29586363f\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=28, eventTime=1756909386, eventType=CREATE_TABLE, dbName=default, tableName=table_d63d2c9adbc5421f9a1c01f637efe55b, message={"server":"","servicePrincipal":"","db":"default","table":"table_d63d2c9adbc5421f9a1c01f637efe55b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756909386},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/warehouse/table_d63d2c9adbc5421f9a1c01f637efe55b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d63d2c9adbc5421f9a1c01f637efe55b\",\"kudu.table_id\":\"6f5c11cb07914fb299328948ea5a0516\",\"transient_lastDdlTime\":\"1756909386\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"2279e9f4777d48f49b3319f959bd577e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756909386,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250903 14:23:28.152118 2446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_867ae72d30fd4ea5b2e85874f80942ad" } new_table_name: "default.Table_90d54f6435f34b629cd27486128824e1" modify_external_catalogs: true
I20250903 14:23:28.152583 2444 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59568:
table { table_name: "default.Table_31acfe71a94349f187dc3fa60a6ecc89" } new_table_name: "default.Table_2c0dd73e42f04addad73b6089d755a89" modify_external_catalogs: true
@ 0x7f304a1d1dcd google::LogMessage::Fail() at ??:0
@ 0x7f304a1d5b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f304a1d17cc google::LogMessage::Flush() at ??:0
@ 0x7f304a1d1a11 google::LogMessage::~LogMessage() at ??:0
I20250903 14:23:28.161523 2580 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Refusing update from remote peer f4c58d929e53460883d05e0f14f20780: Log matching property violated. Preceding OpId in replica: term: 5 index: 92. Preceding OpId from leader: term: 6 index: 94. (index mismatch)
I20250903 14:23:28.162775 2628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
W20250903 14:23:28.167220 2436 consensus_peers.cc:489] T 00000000000000000000000000000000 P f4c58d929e53460883d05e0f14f20780 -> Peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Couldn't send request to peer 2bbe08ab62bf4073825bf1b598ad2edb. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 14:23:28.169175 1067 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
@ 0x7f30606147b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f3060613061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250903 14:23:28.184957 2609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c58d929e53460883d05e0f14f20780. Latest consensus state: current_term: 6 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:28.185302 2609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:28.186187 2570 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:40643; Asking this server to re-register.
@ 0x7f306061acdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250903 14:23:28.187233 1067 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:28.189713 2570 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
@ 0x7f306061ab59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f304cf3a2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f305035e6db start_thread at ??:0
@ 0x7f304908271f clone at ??:0
I20250903 14:23:28.208940 2609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "f4c58d929e53460883d05e0f14f20780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:28.209228 2609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:28.343623 1343 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:28.345512 2570 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:33617; Asking this server to re-register.
I20250903 14:23:28.346367 1343 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:28.348456 2570 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:28.383706 2556 connection.cc:537] server connection from 127.0.0.1:39122 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:28.384644 1354 connection.cc:537] client connection to 127.0.104.61:40437 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 14:23:28.385726 1354 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.61:40437)
W20250903 14:23:28.386211 1354 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:40437)
W20250903 14:23:28.386543 1354 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.61:40437)
W20250903 14:23:28.387012 1354 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:40437)
I20250903 14:23:28.653323 1069 heartbeater.cc:499] Master 127.0.104.61:40437 was elected leader, sending a full tablet report...
I20250903 14:23:28.694346 2609 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:28.694554 2609 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:28.696403 2609 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:28.697402 2555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:28.698678 2555 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:28.698907 2555 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:28.698992 2555 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:28.699302 2609 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250903 14:23:28.701804 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 11: this message will repeat every 5th retry.
I20250903 14:23:29.309391 2660 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:29.309581 2660 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:29.310266 2660 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:29.312235 2555 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:29.312428 2555 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:29.312525 2555 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:29.312774 2660 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:29.974373 2658 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:29.974572 2658 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:29.975205 2658 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:29.976248 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:23:29.976469 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:29.976558 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:29.976841 2658 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:30.186863 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu pid 2534
I20250903 14:23:30.194150 2600 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 14:23:30.308972 2671 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f4c58d929e53460883d05e0f14f20780)
I20250903 14:23:30.309252 2671 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:30.309397 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu with pid 2534
I20250903 14:23:31.069919 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
/tmp/dist-test-tasklEwXqn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--webserver_interface=127.0.104.62
--webserver_port=33279
--builtin_ntp_servers=127.0.104.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 14:23:31.410061 2679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 14:23:31.410405 2679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 14:23:31.410527 2679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 14:23:31.418869 2679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 14:23:31.418987 2679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 14:23:31.419075 2679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 14:23:31.419138 2679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 14:23:31.429872 2679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43435
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=33279
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 14:12:30 UTC on 5fd53c4cbb9d
build id 7880
ASAN enabled
I20250903 14:23:31.431564 2679 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 14:23:31.433347 2679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 14:23:31.442011 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:31.442054 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 14:23:31.442269 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 14:23:31.443037 2679 server_base.cc:1047] running on GCE node
I20250903 14:23:31.443917 2679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 14:23:31.445465 2679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 14:23:31.446700 2679 hybrid_clock.cc:648] HybridClock initialized: now 1756909411446635 us; error 64 us; skew 500 ppm
I20250903 14:23:31.447091 2679 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 14:23:31.449723 2679 webserver.cc:480] Webserver started at http://127.0.104.62:33279/ using document root <none> and password file <none>
I20250903 14:23:31.450726 2679 fs_manager.cc:362] Metadata directory not provided
I20250903 14:23:31.450840 2679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 14:23:31.457094 2679 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250903 14:23:31.460372 2694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 14:23:31.461889 2679 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250903 14:23:31.462106 2679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data,/tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
uuid: "2e83636ce3ca47f3bfc9f84f6755eec2"
format_stamp: "Formatted at 2025-09-03 14:22:58 on dist-test-slave-rxtz"
I20250903 14:23:31.463145 2679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 14:23:31.493484 2679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 14:23:31.494390 2679 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 14:23:31.494668 2679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 14:23:31.514978 2679 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37899
I20250903 14:23:31.514997 2745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37899 every 8 connection(s)
I20250903 14:23:31.516770 2679 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasklEwXqn/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756909361310279-416-0/minicluster-data/master-0/data/info.pb
I20250903 14:23:31.519555 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklEwXqn/build/asan/bin/kudu as pid 2679
I20250903 14:23:31.524612 2746 sys_catalog.cc:263] Verifying existing consensus state
I20250903 14:23:31.528004 2746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap starting.
I20250903 14:23:31.547375 2746 log.cc:826] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Log is configured to *not* fsync() on all Append() calls
W20250903 14:23:31.605731 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 11: this message will repeat every 5th retry.
I20250903 14:23:31.612219 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:31.620764 2746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap replayed 1/1 log segments. Stats: ops{read=94 overwritten=0 applied=94 ignored=0} inserts{seen=62 ignored=0} mutations{seen=73 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 14:23:31.621302 2746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: Bootstrap complete.
I20250903 14:23:31.626654 2746 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:31.627331 2746 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e83636ce3ca47f3bfc9f84f6755eec2, State: Initialized, Role: FOLLOWER
I20250903 14:23:31.627971 2746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 94, Last appended: 6.94, Last appended by leader: 94, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:31.628728 2750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } } }
I20250903 14:23:31.628978 2750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 14:23:31.629695 2746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 14:23:31.646145 2764 catalog_manager.cc:1261] Loaded cluster ID: 2279e9f4777d48f49b3319f959bd577e
I20250903 14:23:31.646339 2764 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: loading cluster ID for follower catalog manager: success
I20250903 14:23:31.653618 2764 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: acquiring CA information for follower catalog manager: success
I20250903 14:23:31.658457 2764 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 14:23:31.656060 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:23:31.662181 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:31.838740 1067 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:31.841030 2711 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" instance_seqno: 1756909380880309) as {username='slave'} at 127.0.104.1:41505; Asking this server to re-register.
I20250903 14:23:31.841643 1067 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:31.843537 2711 ts_manager.cc:194] Registered new tserver with Master: 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849)
I20250903 14:23:31.865366 1343 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37899
I20250903 14:23:31.867113 2711 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" instance_seqno: 1756909381960892) as {username='slave'} at 127.0.104.3:51199; Asking this server to re-register.
I20250903 14:23:31.867776 1343 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:31.869499 2711 ts_manager.cc:194] Registered new tserver with Master: b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
I20250903 14:23:32.045814 2750 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:32.046095 2750 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:32.047932 2750 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:32.048626 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:32.050239 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:32.050486 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:32.050586 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:32.050942 2750 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:32.117861 2711 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" instance_seqno: 1756909381424702) as {username='slave'} at 127.0.104.2:42067; Asking this server to re-register.
I20250903 14:23:32.118723 1204 heartbeater.cc:461] Registering TS with master...
I20250903 14:23:32.120245 2711 ts_manager.cc:194] Registered new tserver with Master: 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
I20250903 14:23:32.800171 2773 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:32.800369 2773 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:32.800982 2773 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:32.803318 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:32.803509 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:32.803613 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:32.803860 2773 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:33.625742 2783 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:33.625964 2783 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:33.626643 2783 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:33.629807 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:33.630013 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:33.630102 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:33.630342 2783 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:33.928655 416 master-stress-test.cc:577] Killing test thread 1/25
I20250903 14:23:35.037791 2798 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:35.037978 2798 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:35.038601 2798 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:35.040966 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:35.041174 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:35.041275 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:35.041556 2798 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:36.665191 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:23:36.665373 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:37.268896 2816 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:37.269138 2816 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:37.269742 2816 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:37.270805 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250903 14:23:37.272028 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:37.272213 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:37.272321 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:37.272559 2816 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:37.553102 2658 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:37.553320 2658 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:37.554165 2658 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:37.555136 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:23:37.555333 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:37.555423 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:37.555732 2658 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:23:38.147991 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 16: this message will repeat every 5th retry.
I20250903 14:23:38.322738 2827 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:38.322964 2827 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:38.323693 2827 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:38.329726 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:38.329929 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:38.330026 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:38.330382 2827 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250903 14:23:41.148816 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 16: this message will repeat every 5th retry.
I20250903 14:23:41.662245 2658 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:41.662485 2658 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:41.663139 2658 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:41.664208 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:41.664456 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:41.664547 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:41.664839 2658 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:41.668332 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:23:41.668520 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:43.032549 2859 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:43.032759 2859 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:43.033468 2859 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:43.034745 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 14:23:43.036540 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:43.036749 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:43.036851 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:43.037163 2859 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:46.671416 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:23:46.671663 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:47.326135 2890 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:47.326433 2890 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:47.327190 2890 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:47.332700 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:47.332877 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:47.332965 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:47.333321 2890 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:47.739187 2877 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:47.739413 2877 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:47.740245 2877 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:47.741349 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:47.741609 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:47.741721 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:47.742043 2877 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:23:48.132398 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 21: this message will repeat every 5th retry.
W20250903 14:23:51.113157 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 21: this message will repeat every 5th retry.
I20250903 14:23:51.521100 2916 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:51.521314 2916 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:23:51.522019 2916 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:23:51.523252 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 14:23:51.524654 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:23:51.524930 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:23:51.525035 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:23:51.525270 2916 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:23:51.674198 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:23:51.674432 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:23:52.948597 2927 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:52.948797 2927 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:23:52.949802 2927 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:23:52.950660 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:23:52.951047 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:23:52.951188 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:52.951496 2927 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:23:56.677030 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:23:56.677263 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:23:57.452399 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 26: this message will repeat every 5th retry.
I20250903 14:23:57.955384 2927 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:23:57.955616 2927 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:23:57.956298 2927 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:23:57.957377 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:23:57.957593 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:23:57.957719 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:23:57.957963 2927 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:24:01.251324 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 26: this message will repeat every 5th retry.
I20250903 14:24:01.680151 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:01.680325 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:24:02.658418 1068 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (110 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:24:02.885267 1205 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (168 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:24:02.932029 1344 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (109 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:24:04.792580 3002 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:04.792796 3002 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:24:04.793507 3002 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:24:04.794669 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:24:04.795985 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:24:04.796236 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:24:04.796342 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:24:04.796583 3002 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:24:06.202899 2967 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:06.203096 2967 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:24:06.203820 2967 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:24:06.204800 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:24:06.204996 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:24:06.205075 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:06.205327 2967 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:06.683187 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:06.683423 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:24:07.287285 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 31: this message will repeat every 5th retry.
I20250903 14:24:08.598707 3024 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:08.598901 3024 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:24:08.599820 3024 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:24:08.600857 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:24:08.601074 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:24:08.601181 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:08.601503 3024 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:10.980998 3035 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:10.981212 3035 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:24:10.981925 3035 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:24:10.983428 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:24:10.985190 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:24:10.985385 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:24:10.985488 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:24:10.985778 3035 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250903 14:24:11.366003 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 31: this message will repeat every 5th retry.
I20250903 14:24:11.686487 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:11.686681 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:24:12.887161 3045 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:12.887391 3045 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:24:12.888087 3045 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:24:12.889098 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:24:12.889402 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:24:12.889518 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:12.889761 3045 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:14.757858 3060 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:14.758085 3060 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:24:14.758795 3060 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:24:14.764608 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:24:14.764824 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
I20250903 14:24:14.764918 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:24:14.765225 3060 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:24:16.689867 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:16.690124 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:24:17.420245 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 36: this message will repeat every 5th retry.
I20250903 14:24:19.430922 3045 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:19.431136 3045 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:24:19.431900 3045 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:24:19.432861 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:24:19.433045 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:24:19.433149 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:19.433423 3045 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:20.470278 3045 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:20.470499 3045 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:24:20.471237 3045 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:24:20.472163 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:24:20.472406 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:24:20.472496 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:20.472769 3045 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:24:21.344770 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 36: this message will repeat every 5th retry.
I20250903 14:24:21.701530 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:21.701946 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:24:24.189421 3045 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:24.189639 3045 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:24:24.190533 3045 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:24:24.191555 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:24:24.191802 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:24:24.191921 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:24.192200 3045 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:26.706036 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:26.706387 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:24:27.104369 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 41: this message will repeat every 5th retry.
W20250903 14:24:29.782775 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 41: this message will repeat every 5th retry.
I20250903 14:24:29.941201 3045 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:29.941414 3045 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:24:29.942050 3045 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:24:29.943482 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:24:29.943743 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:24:29.943847 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:29.944321 3045 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:31.709633 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:31.709873 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:24:32.823880 3178 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:32.824095 3178 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:24:32.825006 3178 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:24:32.826522 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 14:24:32.828006 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:24:32.828245 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:24:32.828347 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:24:32.828697 3178 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:24:36.713248 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:36.713452 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:24:37.249346 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 46: this message will repeat every 5th retry.
W20250903 14:24:39.196260 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 46: this message will repeat every 5th retry.
I20250903 14:24:39.982962 3207 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:39.983283 3207 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:24:39.984519 3207 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:24:39.985577 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:24:39.985821 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:24:39.985935 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:39.986256 3207 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:41.143162 3228 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:41.143419 3228 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:24:41.144587 3228 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:24:41.146126 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:24:41.147863 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:24:41.148154 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:24:41.148273 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:24:41.148633 3228 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:24:41.423547 3207 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:41.423837 3207 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:24:41.424552 3207 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:24:41.425590 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:24:41.425820 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:24:41.425916 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:41.426199 3207 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:41.716310 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:41.716591 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:24:46.719820 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:46.720026 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:24:47.045711 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 51: this message will repeat every 5th retry.
I20250903 14:24:48.637810 3273 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:48.638056 3273 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:24:48.638703 3273 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:24:48.639850 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:24:48.640044 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:24:48.640165 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:48.640460 3273 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:24:49.130564 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 51: this message will repeat every 5th retry.
I20250903 14:24:50.249526 3273 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:50.249761 3273 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:24:50.250399 3273 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:24:50.251585 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:24:50.251828 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:24:50.251919 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:50.252188 3273 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:51.723356 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:51.723659 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:24:53.905618 3312 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:53.905867 3312 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:24:53.906642 3312 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:24:53.908046 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:24:53.909497 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:24:53.909739 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:24:53.909835 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:24:53.910120 3312 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:24:55.610249 3273 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:55.610450 3273 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:24:55.611270 3273 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:24:55.612402 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:24:55.612622 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:24:55.612753 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:55.613029 3273 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:56.480182 3273 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:56.480369 3273 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:24:56.480995 3273 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:24:56.482100 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:24:56.482339 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:24:56.482437 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:56.482777 3273 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:24:56.726624 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:24:56.726817 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:24:57.543893 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 56: this message will repeat every 5th retry.
W20250903 14:24:58.438488 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 56: this message will repeat every 5th retry.
I20250903 14:24:59.244372 3340 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:24:59.244580 3340 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:24:59.245312 3340 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:24:59.246301 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:24:59.246482 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:24:59.246591 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:24:59.246868 3340 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:01.730695 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:01.730898 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:25:02.816432 1068 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (230 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:25:03.102294 1344 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (229 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:25:03.209954 1206 heartbeater.cc:646] Failed to heartbeat to 127.0.104.61:40437 (296 consecutive failures): Network error: Failed to ping master at 127.0.104.61:40437: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
I20250903 14:25:06.734278 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:06.734510 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:25:07.667153 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 61: this message will repeat every 5th retry.
W20250903 14:25:08.636396 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 61: this message will repeat every 5th retry.
I20250903 14:25:09.789508 3411 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:09.789757 3411 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:25:09.790655 3411 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:25:09.792035 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:25:09.797060 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:25:09.797329 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:25:09.797425 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:25:09.797783 3411 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:25:10.788028 3416 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:10.788240 3416 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:25:10.789156 3416 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:25:10.790138 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:25:10.790365 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:25:10.790449 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:10.790766 3416 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:11.737345 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:11.737543 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:25:11.859402 3416 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:11.859669 3416 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:25:11.860339 3416 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:25:11.861565 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:25:11.861771 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:25:11.861889 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:11.862185 3416 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:16.740571 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:16.740778 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:25:17.604703 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 66: this message will repeat every 5th retry.
W20250903 14:25:19.365170 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 66: this message will repeat every 5th retry.
I20250903 14:25:21.744042 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:21.744262 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:25:22.091284 3486 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:22.091533 3486 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:25:22.092300 3486 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:25:22.093353 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:25:22.095028 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:25:22.095229 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:25:22.095338 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:25:22.095679 3486 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:25:26.747599 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:26.747820 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:25:27.528391 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 71: this message will repeat every 5th retry.
W20250903 14:25:28.317625 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 71: this message will repeat every 5th retry.
I20250903 14:25:28.985798 3527 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:28.986029 3527 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:25:28.986800 3527 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:25:28.988054 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:25:28.989571 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:25:28.989784 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:25:28.989873 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:25:28.990188 3527 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:25:30.261144 3519 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:30.261387 3519 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:25:30.262817 3519 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:25:30.263275 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:25:30.263541 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:25:30.263655 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:30.264142 3519 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:31.135699 3519 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:31.135939 3519 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:25:31.136569 3519 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:25:31.137396 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:25:31.137570 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:25:31.137653 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:31.137871 3519 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:31.690423 2700 maintenance_manager.cc:419] P 2e83636ce3ca47f3bfc9f84f6755eec2: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033350
I20250903 14:25:31.749044 2699 maintenance_manager.cc:643] P 2e83636ce3ca47f3bfc9f84f6755eec2: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.058s user 0.026s sys 0.008s Metrics: {"bytes_written":20423,"cfile_init":1,"dirs.queue_time_us":439,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1216,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":762,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":54,"thread_start_us":608,"threads_started":2,"wal-append.queue_time_us":429}
I20250903 14:25:31.750650 2700 maintenance_manager.cc:419] P 2e83636ce3ca47f3bfc9f84f6755eec2: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3596 bytes on disk
I20250903 14:25:31.751263 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:31.751497 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:25:31.751668 2699 maintenance_manager.cc:643] P 2e83636ce3ca47f3bfc9f84f6755eec2: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20250903 14:25:33.715950 3519 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:33.716169 3519 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:25:33.716820 3519 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:25:33.717799 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:25:33.718024 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:25:33.718112 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:33.718410 3519 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:35.160477 3577 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:35.160782 3577 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:25:35.161584 3577 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:25:35.162986 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:25:35.164436 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:25:35.164659 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:25:35.164764 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:25:35.165031 3577 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:25:36.754415 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:36.754585 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:25:36.871327 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 76: this message will repeat every 5th retry.
W20250903 14:25:37.896265 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 76: this message will repeat every 5th retry.
I20250903 14:25:41.757668 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:41.757887 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:25:46.764124 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:46.764356 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:25:47.379222 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 81: this message will repeat every 5th retry.
W20250903 14:25:47.653486 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 81: this message will repeat every 5th retry.
I20250903 14:25:49.208379 3669 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:49.208593 3669 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:25:49.209509 3669 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:25:49.210578 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:25:49.210779 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:25:49.210922 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:49.211236 3669 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:50.546185 3669 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:50.546396 3669 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:25:50.547120 3669 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:25:50.548193 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:25:50.548429 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:25:50.548511 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:50.548815 3669 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:51.767204 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:51.767416 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:25:52.830155 3669 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:52.830343 3669 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:25:52.830935 3669 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:25:52.831918 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:25:52.832108 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:25:52.832247 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:52.832471 3669 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:25:53.313055 3744 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:53.313248 3744 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:25:53.314535 3744 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:25:53.315471 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:25:53.320626 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:25:53.320842 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:25:53.320928 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:25:53.321147 3744 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:25:54.328111 3759 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:54.328297 3759 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:25:54.328914 3759 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:25:54.333345 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:25:54.333556 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:25:54.333663 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:25:54.333927 3759 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250903 14:25:56.200564 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 86: this message will repeat every 5th retry.
I20250903 14:25:56.770323 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:25:56.770493 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:25:57.084672 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 86: this message will repeat every 5th retry.
I20250903 14:25:58.674593 3669 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:25:58.674782 3669 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:25:58.675420 3669 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:25:58.676504 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:25:58.676718 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:25:58.676807 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:25:58.677136 3669 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:26:01.773398 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:01.773577 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:26:02.847044 3669 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:02.847268 3669 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:26:02.847959 3669 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:26:02.848865 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:26:02.849159 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:26:02.849287 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:02.849542 3669 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:26:02.971503 1068 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (350 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:26:03.261528 1344 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (349 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:26:03.659389 1206 heartbeater.cc:646] Failed to heartbeat to 127.0.104.61:40437 (482 consecutive failures): Network error: Failed to ping master at 127.0.104.61:40437: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
I20250903 14:26:06.776314 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:06.776508 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:26:06.960501 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 91: this message will repeat every 5th retry.
W20250903 14:26:07.343371 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 91: this message will repeat every 5th retry.
I20250903 14:26:09.455623 3907 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:09.455843 3907 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:26:09.456703 3907 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:26:09.458702 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 14:26:09.460037 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:26:09.460246 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:26:09.460359 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:26:09.460628 3907 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:26:11.779942 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:11.780176 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:26:12.386202 3927 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:12.386413 3927 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:26:12.387382 3927 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:26:12.388280 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:26:12.388574 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:26:12.388689 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:12.388978 3927 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:26:13.801436 3956 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:13.801680 3956 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:26:13.802510 3956 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:26:13.804953 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:26:13.805217 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:26:13.805296 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:26:13.805559 3956 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:26:15.841575 3927 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:15.841778 3927 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:26:15.842406 3927 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:26:15.843603 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:26:15.843801 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:26:15.843923 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:15.844218 3927 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:26:16.303027 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 96: this message will repeat every 5th retry.
I20250903 14:26:16.523589 3984 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:16.523801 3984 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:26:16.524468 3984 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:26:16.525718 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 14:26:16.527011 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:26:16.527254 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:26:16.527341 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:26:16.527626 3984 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:26:16.782821 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:16.783026 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:26:17.840070 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 96: this message will repeat every 5th retry.
I20250903 14:26:19.745163 4020 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:19.745359 4020 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:26:19.746505 4020 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:26:19.749166 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:26:19.749388 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:26:19.749503 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:26:19.749816 4020 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:26:21.786526 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:21.786774 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:26:26.399338 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 101: this message will repeat every 5th retry.
I20250903 14:26:26.790066 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:26.790266 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:26:27.247545 4063 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:27.247751 4063 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:26:27.248469 4063 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:26:27.249465 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:26:27.249676 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:26:27.249758 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:27.250033 4063 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:26:28.377933 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 101: this message will repeat every 5th retry.
I20250903 14:26:28.383173 4102 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:28.383517 4102 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:26:28.384217 4102 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:26:28.385649 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 14:26:28.387239 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:26:28.387559 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:26:28.387730 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:26:28.388134 4102 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:26:30.983469 4115 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:30.983700 4115 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:26:30.984522 4115 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:26:30.985842 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:26:30.986037 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:26:30.986132 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:30.986375 4115 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:26:31.793284 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:31.793464 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:26:32.229346 4115 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:32.229522 4115 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:26:32.230074 4115 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:26:32.230968 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:26:32.231187 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:26:32.231277 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:32.231511 4115 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:26:36.109511 4115 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:36.109707 4115 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:26:36.110396 4115 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:26:36.111378 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:26:36.111624 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:26:36.111745 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:36.112017 4115 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:26:36.353798 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 106: this message will repeat every 5th retry.
I20250903 14:26:36.796834 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:36.797065 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:26:38.098886 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 106: this message will repeat every 5th retry.
I20250903 14:26:41.057523 4206 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:41.057745 4206 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:26:41.058506 4206 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:26:41.059513 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:26:41.059748 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:26:41.059835 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:41.060138 4206 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:26:41.800045 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:41.800230 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:26:42.782073 4233 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:42.782305 4233 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:26:42.783015 4233 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:26:42.785106 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:26:42.786397 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:26:42.786573 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:26:42.786669 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:26:42.786998 4233 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250903 14:26:46.490437 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 111: this message will repeat every 5th retry.
I20250903 14:26:46.803031 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:46.803210 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:26:47.824079 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 111: this message will repeat every 5th retry.
I20250903 14:26:50.123461 4222 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:50.123667 4222 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:26:50.124539 4222 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:26:50.125381 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:26:50.125561 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:26:50.125679 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:50.125936 4222 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:26:51.806052 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:51.806778 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:26:53.033636 4337 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:53.033852 4337 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:26:53.034516 4337 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:26:53.035791 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:26:53.037170 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:26:53.037377 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:26:53.037496 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:26:53.037818 4337 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:26:54.998217 4222 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:54.998446 4222 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:26:54.999104 4222 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:26:55.000113 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:26:55.000319 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:26:55.000404 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:55.000643 4222 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:26:56.809841 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:26:56.810019 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:26:56.978904 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 116: this message will repeat every 5th retry.
I20250903 14:26:57.413878 4222 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:26:57.414132 4222 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:26:57.414788 4222 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:26:57.415697 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:26:57.415897 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:26:57.416010 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:26:57.416249 4222 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:26:57.538928 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 116: this message will repeat every 5th retry.
I20250903 14:27:01.351958 4399 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:01.352146 4399 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:27:01.352775 4399 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:27:01.353758 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:27:01.353964 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:27:01.354090 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:01.354355 4399 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:27:01.813356 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:01.813557 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:03.126940 1068 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (470 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:27:03.413304 1344 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (469 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:27:03.821970 1206 heartbeater.cc:646] Failed to heartbeat to 127.0.104.61:40437 (668 consecutive failures): Network error: Failed to ping master at 127.0.104.61:40437: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
I20250903 14:27:06.816874 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:06.817101 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:07.035531 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 121: this message will repeat every 5th retry.
W20250903 14:27:07.911039 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 121: this message will repeat every 5th retry.
I20250903 14:27:10.204052 4502 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:10.204339 4502 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:27:10.205475 4502 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:27:10.207223 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:27:10.208818 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:27:10.208984 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:27:10.209106 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:27:10.209368 4502 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:27:11.819880 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:11.820078 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:27:12.054688 4483 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:12.054889 4483 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:27:12.056143 4483 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:27:12.056774 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:27:12.057018 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:27:12.057117 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:12.057322 4483 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:27:13.197698 4483 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:13.197938 4483 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:27:13.198545 4483 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:27:13.199533 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:27:13.199774 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:27:13.199868 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:13.200158 4483 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:27:14.518653 4537 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:14.518899 4537 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:27:14.519546 4537 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:27:14.522269 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:27:14.522519 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:27:14.522615 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:27:14.522923 4537 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:27:16.823343 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:16.823598 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:17.415066 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 126: this message will repeat every 5th retry.
W20250903 14:27:18.267933 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 126: this message will repeat every 5th retry.
I20250903 14:27:21.826738 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:21.826968 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:27:23.318317 4577 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:23.318545 4577 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:27:23.319235 4577 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:27:23.320277 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:27:23.320510 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:27:23.320595 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:23.320822 4577 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:27:26.364382 4577 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:26.364565 4577 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:27:26.365147 4577 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:27:26.366110 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:27:26.366308 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:27:26.366387 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:26.366631 4577 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:27:26.830447 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:26.830804 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:27.446507 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 131: this message will repeat every 5th retry.
W20250903 14:27:28.760411 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 131: this message will repeat every 5th retry.
I20250903 14:27:31.836165 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:31.836341 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:27:32.068681 4662 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:32.068871 4662 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:27:32.069674 4662 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:27:32.070649 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:27:32.070852 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:27:32.070928 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:32.071164 4662 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:27:33.182816 4703 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:33.183555 4703 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:27:33.184561 4703 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:27:33.185859 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 14:27:33.187171 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:27:33.187393 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:27:33.187541 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:27:33.187891 4703 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:27:36.839738 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:36.839958 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:36.869820 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 136: this message will repeat every 5th retry.
W20250903 14:27:39.097833 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 136: this message will repeat every 5th retry.
I20250903 14:27:41.843047 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:41.843256 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:27:42.047264 4791 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:42.047511 4791 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:27:42.048224 4791 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:27:42.049260 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:27:42.050709 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:27:42.050930 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:27:42.051023 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:27:42.051288 4791 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:27:44.237671 4770 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:44.237864 4770 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:27:44.238528 4770 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:27:44.239549 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:27:44.239727 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:27:44.239800 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:44.240027 4770 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:27:45.753561 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 141: this message will repeat every 5th retry.
I20250903 14:27:46.837996 4770 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:46.838160 4770 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:27:46.838811 4770 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:27:46.839589 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:27:46.839787 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:27:46.839867 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:27:46.840065 4770 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:27:46.845773 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:46.845927 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:49.120380 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 141: this message will repeat every 5th retry.
I20250903 14:27:51.375327 4878 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:27:51.375592 4878 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:27:51.376288 4878 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:27:51.377596 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:27:51.378954 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:27:51.379182 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:27:51.379261 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:27:51.379588 4878 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:27:51.848613 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:51.848810 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:56.760562 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 146: this message will repeat every 5th retry.
I20250903 14:27:56.851858 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:27:56.852056 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:27:59.434868 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 146: this message will repeat every 5th retry.
I20250903 14:28:00.037869 4893 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:00.038089 4893 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:28:00.038823 4893 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:28:00.039988 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:28:00.040200 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:28:00.040278 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:28:00.040623 4893 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:28:00.333683 4957 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:00.333945 4957 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:28:00.334761 4957 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:28:00.336170 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:28:00.337715 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:28:00.337980 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:28:00.338068 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:28:00.338411 4957 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:28:01.855065 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:28:01.855297 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:28:03.276397 1068 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (590 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:28:03.568207 1344 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (589 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:28:03.914542 1205 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:43143 (907 consecutive failures): Network error: Failed to ping master at 127.0.104.60:43143: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:28:05.241231 4983 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:05.241422 4983 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:28:05.242056 4983 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:28:05.243079 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:28:05.243232 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:28:05.243288 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:28:05.243511 4983 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:28:06.801105 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 151: this message will repeat every 5th retry.
I20250903 14:28:06.858595 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:28:06.858812 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:28:08.261626 5038 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:08.261843 5038 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:28:08.262563 5038 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:28:08.264014 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:28:08.265563 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:28:08.265959 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:28:08.266099 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:28:08.266414 5038 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250903 14:28:08.910117 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 151: this message will repeat every 5th retry.
I20250903 14:28:11.862095 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:28:11.862326 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:28:12.550904 5072 raft_consensus.cc:491] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:12.551110 5072 raft_consensus.cc:513] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } } peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } }
I20250903 14:28:12.551820 5072 leader_election.cc:290] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863), b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373)
W20250903 14:28:12.552986 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
W20250903 14:28:12.553189 958 leader_election.cc:343] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: f9c4e2c45d19494f86c3c559d3d91202
I20250903 14:28:12.553293 958 leader_election.cc:304] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:28:12.553603 5072 raft_consensus.cc:2747] T f9c4e2c45d19494f86c3c559d3d91202 P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:28:16.549427 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 156: this message will repeat every 5th retry.
I20250903 14:28:16.865535 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:28:16.865722 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 14:28:18.937966 1095 consensus_peers.cc:489] T 05b219b49aa24cbe995c03d1098875dc P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b219b49aa24cbe995c03d1098875dc. This is attempt 156: this message will repeat every 5th retry.
I20250903 14:28:21.869139 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:28:21.869366 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:28:22.382594 5127 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:22.382799 5127 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:28:22.383710 5127 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:28:22.384800 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:28:22.385028 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:28:22.385118 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:28:22.385558 5127 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 14:28:26.766426 1095 consensus_peers.cc:489] T 8ec2abff20f1413fad17fd393d8fd1e9 P 5d4ed06db54c4b5199e190b837bc53fe -> Peer 1adb02d7e9fc4548aa7fd31323ff8ecd (127.0.104.1:39849): Couldn't send request to peer 1adb02d7e9fc4548aa7fd31323ff8ecd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec2abff20f1413fad17fd393d8fd1e9. This is attempt 161: this message will repeat every 5th retry.
I20250903 14:28:26.827030 5189 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:26.827234 5189 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e83636ce3ca47f3bfc9f84f6755eec2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37899 } } peers { permanent_uuid: "f4c58d929e53460883d05e0f14f20780" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40437 } } peers { permanent_uuid: "2bbe08ab62bf4073825bf1b598ad2edb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43143 } }
I20250903 14:28:26.828069 5189 leader_election.cc:290] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437), 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143)
W20250903 14:28:26.829312 2696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 14:28:26.830876 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f4c58d929e53460883d05e0f14f20780 (127.0.104.61:40437): Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111)
W20250903 14:28:26.831115 2696 leader_election.cc:336] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2bbe08ab62bf4073825bf1b598ad2edb (127.0.104.60:43143): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
I20250903 14:28:26.831221 2696 leader_election.cc:304] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e83636ce3ca47f3bfc9f84f6755eec2; no voters: 2bbe08ab62bf4073825bf1b598ad2edb, f4c58d929e53460883d05e0f14f20780
I20250903 14:28:26.831545 5189 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2e83636ce3ca47f3bfc9f84f6755eec2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 14:28:26.872352 2679 master_runner.cc:186] Error getting master registration for 127.0.104.61:40437: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40437: connect: Connection refused (error 111),
I20250903 14:28:26.872557 2679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 14:28:27.356884 5127 raft_consensus.cc:491] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 14:28:27.357086 5127 raft_consensus.cc:513] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1adb02d7e9fc4548aa7fd31323ff8ecd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39849 } } peers { permanent_uuid: "b03995cb49e94f108cf8caa7a502eafe" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43373 } } peers { permanent_uuid: "5d4ed06db54c4b5199e190b837bc53fe" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46863 } }
I20250903 14:28:27.357955 5127 leader_election.cc:290] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373), 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863)
W20250903 14:28:27.359054 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d4ed06db54c4b5199e190b837bc53fe (127.0.104.2:46863): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
W20250903 14:28:27.359236 958 leader_election.cc:343] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b03995cb49e94f108cf8caa7a502eafe (127.0.104.3:43373): Not found: Tablet not found: 29e447cc964646169fbe6ac64090b63e
I20250903 14:28:27.359331 958 leader_election.cc:304] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adb02d7e9fc4548aa7fd31323ff8ecd; no voters: 5d4ed06db54c4b5199e190b837bc53fe, b03995cb49e94f108cf8caa7a502eafe
I20250903 14:28:27.359691 5127 raft_consensus.cc:2747] T 29e447cc964646169fbe6ac64090b63e P 1adb02d7e9fc4548aa7fd31323ff8ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 14:28:28.288528 2700 maintenance_manager.cc:419] P 2e83636ce3ca47f3bfc9f84f6755eec2: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3596 bytes on disk
I20250903 14:28:28.294399 2699 maintenance_manager.cc:643] P 2e83636ce3ca47f3bfc9f84f6755eec2: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.002s sys 0.003s Metrics: {"dirs.queue_time_us":500,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":125,"thread_start_us":695,"threads_started":2,"wal-append.queue_time_us":437}
W20250903 14:28:28.755556 1356 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:28:28.755884 1356 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:28:28.756052 1356 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
W20250903 14:28:28.756196 1356 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
F20250903 14:28:28.767153 1374 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1756909708 (unix time) try "date -d @1756909708" if you are using GNU date ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774057 1369 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663b9a72bf8543eb90c51bd282a2db4d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774057 1369 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663b9a72bf8543eb90c51bd282a2db4d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774123 1383 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774057 1369 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663b9a72bf8543eb90c51bd282a2db4d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774123 1383 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775631 1379 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774057 1369 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663b9a72bf8543eb90c51bd282a2db4d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774123 1383 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775631 1379 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775718 1375 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774057 1369 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663b9a72bf8543eb90c51bd282a2db4d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774123 1383 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775631 1379 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775718 1375 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.776191 1381 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774057 1369 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663b9a72bf8543eb90c51bd282a2db4d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774123 1383 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775631 1379 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775718 1375 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.776191 1381 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.776198 1386 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.776260
*** Check failure stack trace: ***
F20250903 14:28:28.768500 1380 master-stress-test.cc:351] Check failed: _s.ok() 2025status : 0903 14:28:28.768553 1376 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.768868 1378 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.769024 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32a70b2f108542938b8647dc4595cb31 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770146 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770184 1377 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.770393 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b9ce2c993734dcfa81dc61644f332bd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771700 1363 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e3decc796a014477b22f229bd0211387 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771756 1365 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d63d2c9adbc5421f9a1c01f637efe55b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.771993 1367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba72da4f2b84fb39612871a39e8d19f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F2025
03 14:28:28.772006 1382 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773211 1387 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773522 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc1dfd28029e4b3498cd0f42e3227d17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773526 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b56d836b85da4bd4bb8b347606cd1334 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.773864 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20250903 14:28:28.773869 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0b07d90eb033424fab95da7de937a939 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774024 1371 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774057 1369 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_663b9a72bf8543eb90c51bd282a2db4d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.774123 1383 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775631 1379 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.775718 1375 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.776191 1381 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.776198 1386 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)F20250903 14:28:28.776260
1368 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dd32253e3975467098b9075da837375c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:37899,127.0.104.61:40437,127.0.104.60:43143, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43143: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000001a0) received by PID 416 (TID 0x7f88cb5a7700) from PID 416; stack trace: ***
@ 0x7f88e474a980 (unknown) at ??:0
@ 0x7f88dd710fb7 gsignal at ??:0
@ 0x7f88dd712921 abort at ??:0
@ 0x7f88e0448dcd google::LogMessage::Fail() at ??:0
@ 0x7f88e044cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f88e04487cc google::LogMessage::Flush() at ??:0
@ 0x7f88e0449f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40c481 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:328
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f88ddd986df (unknown) at ??:0
@ 0x7f88e473f6db start_thread at ??:0
@ 0x7f88dd7f371f clone at ??:0