Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250831 01:55:06.443040 9405 test_util.cc:276] Using random seed: -1957882635
2025-08-31T01:55:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-31T01:55:06Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskz81FLs/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskz81FLs/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/security.properties
2025-08-31 01:55:12: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskz81FLs/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskz81FLs/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250831 01:55:20.786675 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.126:37509
--webserver_interface=127.9.47.126
--webserver_port=0
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250831 01:55:21.101915 9646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:21.102253 9646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:21.102398 9646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:21.111239 9646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:21.111341 9646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:21.111397 9646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:21.111440 9646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:21.121454 9646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.126:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.47.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:21.123221 9646 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:21.124927 9646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:21.133049 9652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:21.133010 9651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:21.133786 9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:21.134406 9646 server_base.cc:1047] running on GCE node
I20250831 01:55:21.135192 9646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:21.136632 9646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:21.137876 9646 hybrid_clock.cc:648] HybridClock initialized: now 1756605321137830 us; error 40 us; skew 500 ppm
I20250831 01:55:21.138229 9646 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:21.140390 9646 webserver.cc:480] Webserver started at http://127.9.47.126:42371/ using document root <none> and password file <none>
I20250831 01:55:21.141139 9646 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:21.141245 9646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:21.141603 9646 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 01:55:21.145243 9646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data/instance:
uuid: "0b3d4b29236443d8a14d363ec2aab7c8"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:21.146108 9646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/wal/instance:
uuid: "0b3d4b29236443d8a14d363ec2aab7c8"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:21.151330 9646 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250831 01:55:21.154752 9660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:21.156250 9646 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250831 01:55:21.156424 9646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/wal
uuid: "0b3d4b29236443d8a14d363ec2aab7c8"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:21.156623 9646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:21.172947 9646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:21.173880 9646 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:21.174154 9646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:21.193080 9646 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.126:37509
I20250831 01:55:21.193101 9711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.126:37509 every 8 connection(s)
I20250831 01:55:21.194519 9646 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/data/info.pb
I20250831 01:55:21.198065 9712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:21.203614 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 9646
I20250831 01:55:21.203876 9405 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-0/wal/instance
I20250831 01:55:21.203691 9712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } has no permanent_uuid. Determining permanent_uuid...
I20250831 01:55:21.206485 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--webserver_interface=127.9.47.125
--webserver_port=0
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250831 01:55:21.225327 9712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
W20250831 01:55:21.226534 9663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:21.228207 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.125:46791: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:21.277537 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } attempt: 1
W20250831 01:55:21.279860 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.125:46791: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:21.348124 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } attempt: 2
W20250831 01:55:21.349960 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.125:46791: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:21.441216 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } attempt: 3
W20250831 01:55:21.443130 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.125:46791: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:21.529474 9715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:21.529830 9715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:21.529956 9715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:21.538379 9715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:21.538497 9715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:21.538611 9715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:21.538681 9715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:21.548375 9715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.47.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:21.549993 9715 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:21.551589 9715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:21.559027 9721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:21.559182 9722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:21.560387 9724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:21.560842 9715 server_base.cc:1047] running on GCE node
I20250831 01:55:21.561522 9715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:21.562918 9715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:21.564136 9715 hybrid_clock.cc:648] HybridClock initialized: now 1756605321564085 us; error 44 us; skew 500 ppm
I20250831 01:55:21.564455 9715 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:21.566672 9715 webserver.cc:480] Webserver started at http://127.9.47.125:46501/ using document root <none> and password file <none>
I20250831 01:55:21.567378 9715 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:21.567478 9715 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:21.567837 9715 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 01:55:21.570183 9715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/instance:
uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:21.570932 9715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal/instance:
uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:21.576030 9715 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20250831 01:55:21.579181 9730 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:21.580453 9715 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250831 01:55:21.580607 9715 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:21.580811 9715 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:21.586359 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } attempt: 4
W20250831 01:55:21.588124 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.125:46791: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:21.605731 9715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:21.606623 9715 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:21.606871 9715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:21.626848 9715 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.125:46791
I20250831 01:55:21.626856 9781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.125:46791 every 8 connection(s)
I20250831 01:55:21.628288 9715 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
I20250831 01:55:21.631973 9782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:21.635063 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 9715
I20250831 01:55:21.635226 9405 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal/instance
I20250831 01:55:21.636209 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--webserver_interface=127.9.47.124
--webserver_port=0
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250831 01:55:21.637702 9782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } has no permanent_uuid. Determining permanent_uuid...
I20250831 01:55:21.650529 9782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
I20250831 01:55:21.659127 9782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
W20250831 01:55:21.660192 9732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:55:21.661664 9782 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:21.711103 9782 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 1
W20250831 01:55:21.712949 9782 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:21.781185 9782 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 2
W20250831 01:55:21.783200 9782 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:21.874408 9782 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 3
W20250831 01:55:21.876427 9782 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:21.887359 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } attempt: 5
I20250831 01:55:21.893985 9712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
W20250831 01:55:21.895797 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:21.947036 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 1
W20250831 01:55:21.949087 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:55:21.956519 9783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:21.956863 9783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:21.956988 9783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:21.965341 9783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:21.965440 9783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:21.965497 9783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:21.965538 9783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:21.975309 9783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.47.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:21.976920 9783 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:21.978721 9783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:21.987076 9791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:21.986997 9792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:21.987646 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:21.988121 9783 server_base.cc:1047] running on GCE node
I20250831 01:55:21.988822 9783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:21.990202 9783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:21.991397 9783 hybrid_clock.cc:648] HybridClock initialized: now 1756605321991353 us; error 37 us; skew 500 ppm
I20250831 01:55:21.991721 9783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:21.993839 9783 webserver.cc:480] Webserver started at http://127.9.47.124:42935/ using document root <none> and password file <none>
I20250831 01:55:21.994554 9783 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:21.994652 9783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:21.995010 9783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 01:55:21.997323 9783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/instance:
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:21.998100 9783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal/instance:
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:22.003234 9783 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250831 01:55:22.006629 9800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:22.007957 9783 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250831 01:55:22.008111 9783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:22.008299 9783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:22.017295 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 2
W20250831 01:55:22.019016 9712 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:22.019636 9782 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 4
W20250831 01:55:22.021116 9782 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.9.47.124:37481: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:22.025609 9783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:22.026464 9783 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:22.026715 9783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:22.046303 9783 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.124:37481
I20250831 01:55:22.046327 9851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.124:37481 every 8 connection(s)
I20250831 01:55:22.047731 9783 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
I20250831 01:55:22.051242 9852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:22.052781 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 9783
I20250831 01:55:22.053028 9405 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal/instance
I20250831 01:55:22.058775 9852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } has no permanent_uuid. Determining permanent_uuid...
I20250831 01:55:22.080896 9852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
I20250831 01:55:22.087756 9852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
I20250831 01:55:22.098256 9852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap starting.
I20250831 01:55:22.101568 9852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:22.102658 9852 log.cc:826] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:22.105436 9852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: No bootstrap required, opened a new log
I20250831 01:55:22.110316 9852 raft_consensus.cc:357] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.110641 9852 raft_consensus.cc:383] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:22.110745 9852 raft_consensus.cc:738] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Initialized, Role: FOLLOWER
I20250831 01:55:22.111369 9852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.112624 9860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.112964 9860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:22.113617 9852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:22.125221 9712 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 3
W20250831 01:55:22.126745 9874 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250831 01:55:22.126979 9874 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250831 01:55:22.122490 9783 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: OK, instance_id { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" instance_seqno: 1756605321188125 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250831 01:55:22.133140 9783 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:22.137106 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Bootstrap starting.
I20250831 01:55:22.140645 9712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:22.142191 9712 log.cc:826] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:22.145530 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: No bootstrap required, opened a new log
I20250831 01:55:22.152698 9712 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.153172 9712 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:22.153337 9712 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3d4b29236443d8a14d363ec2aab7c8, State: Initialized, Role: FOLLOWER
I20250831 01:55:22.154091 9712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.155449 9878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.155817 9878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:22.156725 9712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: configured and running, proceeding with master startup.
W20250831 01:55:22.176781 9666 tablet.cc:2378] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250831 01:55:22.278774 9806 tablet.cc:2378] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 01:55:22.298516 9646 master_runner.cc:186] Error getting master registration for 127.9.47.125:46791: OK, instance_id { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" instance_seqno: 1756605321621366 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250831 01:55:22.306529 9646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:55:22.309815 9893 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250831 01:55:22.309948 9893 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250831 01:55:22.320319 9782 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } attempt: 5
I20250831 01:55:22.335134 9782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap starting.
I20250831 01:55:22.339071 9782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:22.340444 9782 log.cc:826] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:22.350795 9782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: No bootstrap required, opened a new log
I20250831 01:55:22.355546 9782 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.355834 9782 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:22.355964 9782 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3cdeb22a7546b988ab9938f8dc1027, State: Initialized, Role: FOLLOWER
I20250831 01:55:22.356572 9782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.357491 9896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.357883 9896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:22.358913 9782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: configured and running, proceeding with master startup.
W20250831 01:55:22.359112 9736 tablet.cc:2378] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 01:55:22.399909 9715 master_runner.cc:263] No leader master found from master 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:22.400060 9715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:55:22.409348 9911 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250831 01:55:22.409637 9911 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250831 01:55:22.532107 9878 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:22.532414 9878 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.535045 9878 leader_election.cc:290] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:22.536199 9757 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" is_pre_election: true
I20250831 01:55:22.536832 9757 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b3d4b29236443d8a14d363ec2aab7c8 in term 0.
I20250831 01:55:22.542382 9663 leader_election.cc:304] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027; no voters:
I20250831 01:55:22.543800 9827 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905" is_pre_election: true
I20250831 01:55:22.544305 9827 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b3d4b29236443d8a14d363ec2aab7c8 in term 0.
I20250831 01:55:22.545305 9878 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:22.545466 9878 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:22.545562 9878 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:22.551507 9878 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.563035 9757 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
I20250831 01:55:22.563262 9757 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:22.571743 9757 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b3d4b29236443d8a14d363ec2aab7c8 in term 1.
I20250831 01:55:22.572453 9663 leader_election.cc:304] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027; no voters:
I20250831 01:55:22.581193 9827 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905"
I20250831 01:55:22.581434 9827 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:22.588259 9827 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b3d4b29236443d8a14d363ec2aab7c8 in term 1.
I20250831 01:55:22.590900 9878 leader_election.cc:290] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 1 election: Requested vote from peers 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:22.591395 9878 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:22.591910 9878 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0b3d4b29236443d8a14d363ec2aab7c8, State: Running, Role: LEADER
I20250831 01:55:22.592911 9878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:22.605499 9913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3d4b29236443d8a14d363ec2aab7c8. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.610298 9913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:22.618510 9915 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 01:55:22.620769 9915 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 01:55:22.642766 9827 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Refusing update from remote peer 0b3d4b29236443d8a14d363ec2aab7c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:22.643770 9913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:22.645900 9757 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Refusing update from remote peer 0b3d4b29236443d8a14d363ec2aab7c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:22.646955 9878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:22.669103 9896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3d4b29236443d8a14d363ec2aab7c8. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.669379 9896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:22.672832 9916 mvcc.cc:204] Tried to move back new op lower bound from 7195055401533435904 to 7195055401359446016. Current Snapshot: MvccSnapshot[applied={T|T < 7195055401533435904}]
I20250831 01:55:22.674935 9918 mvcc.cc:204] Tried to move back new op lower bound from 7195055401533435904 to 7195055401359446016. Current Snapshot: MvccSnapshot[applied={T|T < 7195055401533435904}]
I20250831 01:55:22.686309 9913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.686560 9913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:22.688860 9917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3d4b29236443d8a14d363ec2aab7c8. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.689160 9917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:22.692704 9878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.693470 9878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:22.695572 9920 mvcc.cc:204] Tried to move back new op lower bound from 7195055401533435904 to 7195055401359446016. Current Snapshot: MvccSnapshot[applied={T|T < 7195055401533435904}]
I20250831 01:55:22.698607 9915 catalog_manager.cc:1349] Generated new cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:22.699066 9915 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 01:55:22.709220 9917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.709491 9917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:22.710333 9896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:22.710538 9896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:22.723845 9915 catalog_manager.cc:1372] Generated new certificate authority record
I20250831 01:55:22.725646 9915 catalog_manager.cc:1506] Loading token signing keys...
I20250831 01:55:22.742807 9915 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Generated new TSK 0
I20250831 01:55:22.744793 9915 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 01:55:22.745750 9915 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250831 01:55:22.746587 9915 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250831 01:55:22.762360 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.47.65:0
--local_ip_for_outbound_sockets=127.9.47.65
--webserver_interface=127.9.47.65
--webserver_port=0
--tserver_master_addrs=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250831 01:55:23.085928 9926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:23.086269 9926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:23.086423 9926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:23.094558 9926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:23.094803 9926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.47.65
I20250831 01:55:23.104328 9926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.47.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.47.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.47.65
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:23.105957 9926 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:23.107651 9926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:23.116897 9932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:23.116958 9931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:23.117628 9934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:23.118270 9926 server_base.cc:1047] running on GCE node
I20250831 01:55:23.118849 9926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:23.119894 9926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:23.123193 9926 hybrid_clock.cc:648] HybridClock initialized: now 1756605323123145 us; error 35 us; skew 500 ppm
I20250831 01:55:23.123473 9926 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:23.125496 9926 webserver.cc:480] Webserver started at http://127.9.47.65:39141/ using document root <none> and password file <none>
I20250831 01:55:23.126222 9926 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:23.126305 9926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:23.126619 9926 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 01:55:23.128780 9874 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:23.128856 9874 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: loading cluster ID for follower catalog manager: success
I20250831 01:55:23.128950 9926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data/instance:
uuid: "67e9e49e7a864057adb3e685607c9844"
format_stamp: "Formatted at 2025-08-31 01:55:23 on dist-test-slave-gcjs"
I20250831 01:55:23.129691 9926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/instance:
uuid: "67e9e49e7a864057adb3e685607c9844"
format_stamp: "Formatted at 2025-08-31 01:55:23 on dist-test-slave-gcjs"
I20250831 01:55:23.131341 9874 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: acquiring CA information for follower catalog manager: success
I20250831 01:55:23.132933 9874 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:23.134578 9926 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20250831 01:55:23.137681 9940 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:23.138882 9926 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 01:55:23.139004 9926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal
uuid: "67e9e49e7a864057adb3e685607c9844"
format_stamp: "Formatted at 2025-08-31 01:55:23 on dist-test-slave-gcjs"
I20250831 01:55:23.139166 9926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:23.160080 9926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:23.160960 9926 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:23.161185 9926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:23.162117 9926 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 01:55:23.163694 9926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 01:55:23.163785 9926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:23.163863 9926 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 01:55:23.163946 9926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:23.197537 9926 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.65:43439
I20250831 01:55:23.197602 10052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.65:43439 every 8 connection(s)
I20250831 01:55:23.198973 9926 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/data/info.pb
I20250831 01:55:23.209348 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 9926
I20250831 01:55:23.209637 9405 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/instance
I20250831 01:55:23.214983 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.47.66:0
--local_ip_for_outbound_sockets=127.9.47.66
--webserver_interface=127.9.47.66
--webserver_port=0
--tserver_master_addrs=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250831 01:55:23.238476 10054 heartbeater.cc:344] Connected to a master server at 127.9.47.126:37509
I20250831 01:55:23.238884 10054 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:23.238956 10053 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:23.239089 10053 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:23.239979 10054 heartbeater.cc:507] Master 127.9.47.126:37509 requested a full tablet report, sending...
I20250831 01:55:23.240151 10053 heartbeater.cc:507] Master 127.9.47.124:37481 requested a full tablet report, sending...
I20250831 01:55:23.242777 9677 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:23.243522 9817 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:23.244760 9677 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.47.65:52921
I20250831 01:55:23.245368 10055 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:23.245510 10055 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:23.246147 10055 heartbeater.cc:507] Master 127.9.47.125:46791 requested a full tablet report, sending...
I20250831 01:55:23.247879 9747 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:23.411546 9911 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:23.411650 9911 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: loading cluster ID for follower catalog manager: success
I20250831 01:55:23.414402 9911 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: acquiring CA information for follower catalog manager: success
I20250831 01:55:23.416119 9911 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250831 01:55:23.572901 10063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:23.573259 10063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:23.573444 10063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:23.582093 10063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:23.582357 10063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.47.66
I20250831 01:55:23.592231 10063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.47.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.47.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.47.66
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:23.593935 10063 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:23.595784 10063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:23.605440 10069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:23.605490 10068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:23.606149 10071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:23.606491 10063 server_base.cc:1047] running on GCE node
I20250831 01:55:23.607091 10063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:23.608026 10063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:23.609248 10063 hybrid_clock.cc:648] HybridClock initialized: now 1756605323609206 us; error 30 us; skew 500 ppm
I20250831 01:55:23.609633 10063 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:23.612049 10063 webserver.cc:480] Webserver started at http://127.9.47.66:41513/ using document root <none> and password file <none>
I20250831 01:55:23.612843 10063 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:23.612954 10063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:23.613340 10063 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 01:55:23.616099 10063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data/instance:
uuid: "fbe14365c42e4c6ba1173d829f997f81"
format_stamp: "Formatted at 2025-08-31 01:55:23 on dist-test-slave-gcjs"
I20250831 01:55:23.616925 10063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/instance:
uuid: "fbe14365c42e4c6ba1173d829f997f81"
format_stamp: "Formatted at 2025-08-31 01:55:23 on dist-test-slave-gcjs"
I20250831 01:55:23.621341 10076 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b3d4b29236443d8a14d363ec2aab7c8)
I20250831 01:55:23.621539 10076 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:23.622757 10076 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:23.623016 10063 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250831 01:55:23.623417 9827 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905" is_pre_election: true
I20250831 01:55:23.623551 9687 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" is_pre_election: true
I20250831 01:55:23.624325 9731 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:23.624650 10076 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 01:55:23.630471 10078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:23.631889 10063 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.004s
I20250831 01:55:23.632037 10063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal
uuid: "fbe14365c42e4c6ba1173d829f997f81"
format_stamp: "Formatted at 2025-08-31 01:55:23 on dist-test-slave-gcjs"
I20250831 01:55:23.632251 10063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:23.657794 10063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:23.658726 10063 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:23.659011 10063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:23.660006 10063 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 01:55:23.661825 10063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 01:55:23.661928 10063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250831 01:55:23.662052 10063 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 01:55:23.662134 10063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:23.697041 10063 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.66:46085
I20250831 01:55:23.697110 10190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.66:46085 every 8 connection(s)
I20250831 01:55:23.698449 10063 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/data/info.pb
I20250831 01:55:23.701406 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 10063
I20250831 01:55:23.701699 9405 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/instance
I20250831 01:55:23.708113 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.47.67:0
--local_ip_for_outbound_sockets=127.9.47.67
--webserver_interface=127.9.47.67
--webserver_port=0
--tserver_master_addrs=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250831 01:55:23.716512 10191 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:23.716967 10191 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:23.717248 10193 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:23.717373 10193 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:23.718425 10191 heartbeater.cc:507] Master 127.9.47.124:37481 requested a full tablet report, sending...
I20250831 01:55:23.718454 10193 heartbeater.cc:507] Master 127.9.47.125:46791 requested a full tablet report, sending...
I20250831 01:55:23.720393 9747 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:23.720552 9817 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:23.721940 10192 heartbeater.cc:344] Connected to a master server at 127.9.47.126:37509
I20250831 01:55:23.722069 10192 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:23.722470 10192 heartbeater.cc:507] Master 127.9.47.126:37509 requested a full tablet report, sending...
I20250831 01:55:23.723318 9677 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:23.724097 9677 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.47.66:36923
I20250831 01:55:23.747273 10054 heartbeater.cc:499] Master 127.9.47.126:37509 was elected leader, sending a full tablet report...
W20250831 01:55:24.039590 10198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:24.040066 10198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:24.040243 10198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:24.051292 10198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:24.051548 10198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.47.67
I20250831 01:55:24.069669 10198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.47.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.47.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.47.67
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:24.072100 10198 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:24.075114 10198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:24.085515 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:24.085433 10204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:24.086352 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:24.086963 10198 server_base.cc:1047] running on GCE node
I20250831 01:55:24.087507 10198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:24.088413 10198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:24.089629 10198 hybrid_clock.cc:648] HybridClock initialized: now 1756605324089591 us; error 44 us; skew 500 ppm
I20250831 01:55:24.089972 10198 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:24.092135 10198 webserver.cc:480] Webserver started at http://127.9.47.67:37163/ using document root <none> and password file <none>
I20250831 01:55:24.092890 10198 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:24.092998 10198 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:24.093351 10198 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 01:55:24.096019 10198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data/instance:
uuid: "903e5d68fe5e4988a6b54623da741af5"
format_stamp: "Formatted at 2025-08-31 01:55:24 on dist-test-slave-gcjs"
I20250831 01:55:24.096833 10198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/instance:
uuid: "903e5d68fe5e4988a6b54623da741af5"
format_stamp: "Formatted at 2025-08-31 01:55:24 on dist-test-slave-gcjs"
I20250831 01:55:24.102378 10198 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250831 01:55:24.106042 10212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:24.107589 10198 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250831 01:55:24.107784 10198 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal
uuid: "903e5d68fe5e4988a6b54623da741af5"
format_stamp: "Formatted at 2025-08-31 01:55:24 on dist-test-slave-gcjs"
I20250831 01:55:24.108001 10198 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:24.131935 10198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:24.132788 10198 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:24.133044 10198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:24.134027 10198 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 01:55:24.135697 10198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 01:55:24.135787 10198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:24.135901 10198 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 01:55:24.135974 10198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:24.168687 10198 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.67:33173
I20250831 01:55:24.168758 10324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.67:33173 every 8 connection(s)
I20250831 01:55:24.170153 10198 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/data/info.pb
I20250831 01:55:24.172048 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 10198
I20250831 01:55:24.172241 9405 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/instance
I20250831 01:55:24.183231 10329 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b3d4b29236443d8a14d363ec2aab7c8)
I20250831 01:55:24.183619 10329 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:24.194892 9687 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" is_pre_election: true
I20250831 01:55:24.196283 9756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" is_pre_election: true
I20250831 01:55:24.197055 9803 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:24.198999 10329 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
I20250831 01:55:24.199232 10329 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 01:55:24.201570 10327 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:24.206138 10327 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:24.207266 10327 heartbeater.cc:507] Master 127.9.47.125:46791 requested a full tablet report, sending...
I20250831 01:55:24.208101 10326 heartbeater.cc:344] Connected to a master server at 127.9.47.126:37509
I20250831 01:55:24.208235 10326 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:24.208760 10326 heartbeater.cc:507] Master 127.9.47.126:37509 requested a full tablet report, sending...
I20250831 01:55:24.208851 10325 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:24.208971 10325 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:24.209347 9746 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:24.209451 10325 heartbeater.cc:507] Master 127.9.47.124:37481 requested a full tablet report, sending...
I20250831 01:55:24.209658 9676 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:24.210801 9816 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:24.210835 9676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.47.67:36619
I20250831 01:55:24.225922 9405 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250831 01:55:24.226152 10192 heartbeater.cc:499] Master 127.9.47.126:37509 was elected leader, sending a full tablet report...
I20250831 01:55:24.260690 9677 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_0c8cd6f43d2e4afeaf6302f110ebac45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.260725 9676 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_e8582bab06b043f79df730d33ed076fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.261328 9675 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.261981 9674 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_547c8e6d3c104af9b87f1ae477d22ee0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.263233 9673 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_cd781cda81e34e17b5cc76c5689b9600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.266861 9672 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_e32aefba1e29406f91d7559a02f1323f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.267454 9671 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_519e74debc8b417c8fe427b8b4c3bd29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 01:55:24.269222 9677 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0c8cd6f43d2e4afeaf6302f110ebac45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 01:55:24.273233 9670 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_b0242810ea7a4fe3a1d3d8431eca51d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.274250 9669 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_54383364d8c14085a4963804ac2d6ba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 01:55:24.274946 9674 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_547c8e6d3c104af9b87f1ae477d22ee0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:24.274946 9675 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:24.276082 9676 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e8582bab06b043f79df730d33ed076fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:24.276994 9673 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cd781cda81e34e17b5cc76c5689b9600 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:24.278040 9672 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e32aefba1e29406f91d7559a02f1323f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 01:55:24.280057 9668 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_2faa27c5430c42e7b69a131d36238fdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:24.286370 9405 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskz81FLs/build/asan/bin/kudu pid 9715
I20250831 01:55:24.288057 9777 generic_service.cc:196] Checking for leaks (request via RPC)
I20250831 01:55:24.449545 9405 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskz81FLs/build/asan/bin/kudu with pid 9715
W20250831 01:55:24.499536 9663 consensus_peers.cc:489] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 -> Peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Couldn't send request to peer 6b3cdeb22a7546b988ab9938f8dc1027. Status: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:24.712875 10326 heartbeater.cc:499] Master 127.9.47.126:37509 was elected leader, sending a full tablet report...
W20250831 01:55:24.723608 10327 heartbeater.cc:646] Failed to heartbeat to 127.9.47.125:46791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:24.724798 10193 heartbeater.cc:646] Failed to heartbeat to 127.9.47.125:46791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:24.732650 9677 catalog_manager.cc:2149] created HMS catalog entry for table [id=71f91154db86487a915ab787c1495178]
W20250831 01:55:24.758860 10055 heartbeater.cc:646] Failed to heartbeat to 127.9.47.125:46791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:24.780651 9674 catalog_manager.cc:2149] created HMS catalog entry for table [id=b9f98af128a1402ca281bd1f8853c194]
I20250831 01:55:24.780733 9677 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_0c8cd6f43d2e4afeaf6302f110ebac45" } new_table_name: "default.Table_3bbd369a7fa546ba9996f1935a35e1cd" modify_external_catalogs: true
W20250831 01:55:24.785575 9670 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:24.785575 9671 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_519e74debc8b417c8fe427b8b4c3bd29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 01:55:24.800302 9674 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_547c8e6d3c104af9b87f1ae477d22ee0" } new_table_name: "default.Table_03598ee84555460b9138a48936385e5e" modify_external_catalogs: true
I20250831 01:55:24.824621 9675 catalog_manager.cc:2149] created HMS catalog entry for table [id=9584cc0bbc7e437e967c3b8c39ab609b]
I20250831 01:55:24.850703 9675 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5" } new_table_name: "default.Table_513033900eee43b68b82843e69d0bb74" modify_external_catalogs: true
I20250831 01:55:24.860343 10126 tablet_service.cc:1468] Processing CreateTablet for tablet 7080923b09554076b43599fe5504cfb8 (DEFAULT_TABLE table=default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.861243 10125 tablet_service.cc:1468] Processing CreateTablet for tablet b61feb48c6164268a2d0b7c892ab486a (DEFAULT_TABLE table=default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.861268 10124 tablet_service.cc:1468] Processing CreateTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 (DEFAULT_TABLE table=default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.861907 10126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7080923b09554076b43599fe5504cfb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.862344 10124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.862627 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61feb48c6164268a2d0b7c892ab486a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.866906 9676 catalog_manager.cc:2149] created HMS catalog entry for table [id=b90d279f47a141ed846c461cd398d5e7]
I20250831 01:55:24.878368 10260 tablet_service.cc:1468] Processing CreateTablet for tablet 7080923b09554076b43599fe5504cfb8 (DEFAULT_TABLE table=default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.880184 10259 tablet_service.cc:1468] Processing CreateTablet for tablet b61feb48c6164268a2d0b7c892ab486a (DEFAULT_TABLE table=default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.881246 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61feb48c6164268a2d0b7c892ab486a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.882014 10258 tablet_service.cc:1468] Processing CreateTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 (DEFAULT_TABLE table=default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.882477 10258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.884619 10257 tablet_service.cc:1468] Processing CreateTablet for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (DEFAULT_TABLE table=default.table_e8582bab06b043f79df730d33ed076fa [id=b90d279f47a141ed846c461cd398d5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.885105 10257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe4c7b8517747a9a9baef5c25d314f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.886122 9676 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_e8582bab06b043f79df730d33ed076fa" } new_table_name: "default.Table_037da9465b8a41fda06bdeb1cf150d8b" modify_external_catalogs: true
I20250831 01:55:24.886454 10123 tablet_service.cc:1468] Processing CreateTablet for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (DEFAULT_TABLE table=default.table_e8582bab06b043f79df730d33ed076fa [id=b90d279f47a141ed846c461cd398d5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.886950 10123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe4c7b8517747a9a9baef5c25d314f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.887167 10260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7080923b09554076b43599fe5504cfb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.903602 10395 tablet_bootstrap.cc:492] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:24.907274 10395 tablet_bootstrap.cc:654] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:24.907626 9987 tablet_service.cc:1468] Processing CreateTablet for tablet b61feb48c6164268a2d0b7c892ab486a (DEFAULT_TABLE table=default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.908615 10395 log.cc:826] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:24.909165 9987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61feb48c6164268a2d0b7c892ab486a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.910164 9985 tablet_service.cc:1468] Processing CreateTablet for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (DEFAULT_TABLE table=default.table_e8582bab06b043f79df730d33ed076fa [id=b90d279f47a141ed846c461cd398d5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.910579 9985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe4c7b8517747a9a9baef5c25d314f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.911801 10395 tablet_bootstrap.cc:492] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:24.912042 10395 ts_tablet_manager.cc:1397] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250831 01:55:24.914690 9988 tablet_service.cc:1468] Processing CreateTablet for tablet 7080923b09554076b43599fe5504cfb8 (DEFAULT_TABLE table=default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.915380 9986 tablet_service.cc:1468] Processing CreateTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 (DEFAULT_TABLE table=default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.916206 9988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7080923b09554076b43599fe5504cfb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.918948 10395 raft_consensus.cc:357] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:24.919323 10395 raft_consensus.cc:383] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:24.919428 10395 raft_consensus.cc:738] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:24.920172 10395 consensus_queue.cc:260] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:24.922142 9986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.932516 9673 catalog_manager.cc:2149] created HMS catalog entry for table [id=ad1bd5ef25a343f0ad51d4e99ec44792]
I20250831 01:55:24.939448 9673 master_service.cc:948] ReplaceTablet: received request to replace tablet b61feb48c6164268a2d0b7c892ab486a from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:24.940919 10395 ts_tablet_manager.cc:1428] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.029s user 0.009s sys 0.007s
I20250831 01:55:24.947151 10385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:24.947407 10385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:24.948411 10395 tablet_bootstrap.cc:492] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:24.950438 10395 tablet_bootstrap.cc:654] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
W20250831 01:55:24.951989 10194 tablet.cc:2378] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 01:55:24.954114 9673 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Sending DeleteTablet for 3 replicas of tablet b61feb48c6164268a2d0b7c892ab486a
I20250831 01:55:24.954774 9673 catalog_manager.cc:6652] ReplaceTablet: tablet b61feb48c6164268a2d0b7c892ab486a deleted and replaced by tablet 3772a8213fd74d1a995e0c8eed42c08b
I20250831 01:55:24.957401 9673 master_service.cc:948] ReplaceTablet: received request to replace tablet 7080923b09554076b43599fe5504cfb8 from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:24.958242 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet b61feb48c6164268a2d0b7c892ab486a with delete_type TABLET_DATA_DELETED (replaced by tablet 3772a8213fd74d1a995e0c8eed42c08b) from {username='slave'} at 127.0.0.1:42118
I20250831 01:55:24.959159 10395 tablet_bootstrap.cc:492] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:24.959374 10395 ts_tablet_manager.cc:1397] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250831 01:55:24.960647 10395 raft_consensus.cc:357] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:24.960914 10395 raft_consensus.cc:383] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:24.961001 10395 raft_consensus.cc:738] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:24.961314 10395 consensus_queue.cc:260] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:24.962754 10395 ts_tablet_manager.cc:1428] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250831 01:55:24.962674 10256 tablet_service.cc:1515] Processing DeleteTablet for tablet b61feb48c6164268a2d0b7c892ab486a with delete_type TABLET_DATA_DELETED (replaced by tablet 3772a8213fd74d1a995e0c8eed42c08b) from {username='slave'} at 127.0.0.1:48110
I20250831 01:55:24.963145 10395 tablet_bootstrap.cc:492] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:24.965027 10395 tablet_bootstrap.cc:654] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:24.967816 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet b61feb48c6164268a2d0b7c892ab486a with delete_type TABLET_DATA_DELETED (replaced by tablet 3772a8213fd74d1a995e0c8eed42c08b) from {username='slave'} at 127.0.0.1:38376
I20250831 01:55:24.978888 10126 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbcda68336141109d5e3c96f97454a5 (DEFAULT_TABLE table=default.table_cd781cda81e34e17b5cc76c5689b9600 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.979481 10126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbcda68336141109d5e3c96f97454a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.980850 10397 raft_consensus.cc:491] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:24.981014 10260 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbcda68336141109d5e3c96f97454a5 (DEFAULT_TABLE table=default.table_cd781cda81e34e17b5cc76c5689b9600 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.981114 10397 raft_consensus.cc:513] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:24.981446 10260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbcda68336141109d5e3c96f97454a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.983230 10397 leader_election.cc:290] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:24.984757 9988 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbcda68336141109d5e3c96f97454a5 (DEFAULT_TABLE table=default.table_cd781cda81e34e17b5cc76c5689b9600 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:24.985185 9988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbcda68336141109d5e3c96f97454a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:24.992182 10405 tablet_bootstrap.cc:492] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:24.994722 10400 tablet_replica.cc:331] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:24.995247 10400 raft_consensus.cc:2241] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:24.995680 10400 raft_consensus.cc:2270] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.002476 10400 ts_tablet_manager.cc:1905] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.003979 10395 tablet_bootstrap.cc:492] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.004135 10395 ts_tablet_manager.cc:1397] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.006s
I20250831 01:55:25.004972 10395 raft_consensus.cc:357] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.005162 10395 raft_consensus.cc:383] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.005262 10395 raft_consensus.cc:738] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.004765 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61feb48c6164268a2d0b7c892ab486a" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.005456 10395 consensus_queue.cc:260] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.007530 10395 ts_tablet_manager.cc:1428] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250831 01:55:25.008093 9987 tablet_service.cc:1468] Processing CreateTablet for tablet 3772a8213fd74d1a995e0c8eed42c08b (DEFAULT_TABLE table=default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.009434 10395 tablet_bootstrap.cc:492] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.010922 9673 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Sending DeleteTablet for 3 replicas of tablet 7080923b09554076b43599fe5504cfb8
I20250831 01:55:25.011269 9987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3772a8213fd74d1a995e0c8eed42c08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.011463 10395 tablet_bootstrap.cc:654] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.012253 10259 tablet_service.cc:1515] Processing DeleteTablet for tablet 7080923b09554076b43599fe5504cfb8 with delete_type TABLET_DATA_DELETED (replaced by tablet fdf49ea1050b4326b1df977f021d3d31) from {username='slave'} at 127.0.0.1:48110
I20250831 01:55:25.012831 9673 catalog_manager.cc:6652] ReplaceTablet: tablet 7080923b09554076b43599fe5504cfb8 deleted and replaced by tablet fdf49ea1050b4326b1df977f021d3d31
I20250831 01:55:25.009135 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61feb48c6164268a2d0b7c892ab486a" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.014336 9672 catalog_manager.cc:2149] created HMS catalog entry for table [id=0c52c96929564376a2419751be04d5b8]
I20250831 01:55:25.018193 10258 tablet_service.cc:1468] Processing CreateTablet for tablet 3772a8213fd74d1a995e0c8eed42c08b (DEFAULT_TABLE table=default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.018193 10126 tablet_service.cc:1468] Processing CreateTablet for tablet 3772a8213fd74d1a995e0c8eed42c08b (DEFAULT_TABLE table=default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.018646 10258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3772a8213fd74d1a995e0c8eed42c08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.020750 10126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3772a8213fd74d1a995e0c8eed42c08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.024825 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet 7080923b09554076b43599fe5504cfb8 with delete_type TABLET_DATA_DELETED (replaced by tablet fdf49ea1050b4326b1df977f021d3d31) from {username='slave'} at 127.0.0.1:42118
W20250831 01:55:25.025096 10080 leader_election.cc:343] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.025911 10395 tablet_bootstrap.cc:492] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.026141 10395 ts_tablet_manager.cc:1397] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250831 01:55:25.026947 10395 raft_consensus.cc:357] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.027206 10395 raft_consensus.cc:383] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.027321 10395 raft_consensus.cc:738] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.027612 10395 consensus_queue.cc:260] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.028193 10395 ts_tablet_manager.cc:1428] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250831 01:55:25.028312 9672 master_service.cc:948] ReplaceTablet: received request to replace tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:25.029934 10395 tablet_bootstrap.cc:492] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.031591 10395 tablet_bootstrap.cc:654] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.033214 10260 tablet_service.cc:1468] Processing CreateTablet for tablet fdf49ea1050b4326b1df977f021d3d31 (DEFAULT_TABLE table=default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.034540 10125 tablet_service.cc:1468] Processing CreateTablet for tablet fdf49ea1050b4326b1df977f021d3d31 (DEFAULT_TABLE table=default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.034994 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf49ea1050b4326b1df977f021d3d31. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 01:55:25.036345 9669 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_54383364d8c14085a4963804ac2d6ba0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:25.037501 9668 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2faa27c5430c42e7b69a131d36238fdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 01:55:25.040042 9672 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Sending DeleteTablet for 3 replicas of tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0
I20250831 01:55:25.040578 9672 catalog_manager.cc:6652] ReplaceTablet: tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 deleted and replaced by tablet 1def9b41fbe94b4ba680d0d4afd0d2b8
I20250831 01:55:25.040814 9672 master_service.cc:948] ReplaceTablet: received request to replace tablet 7080923b09554076b43599fe5504cfb8 from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:25.041285 9672 master_service.cc:948] ReplaceTablet: received request to replace tablet b61feb48c6164268a2d0b7c892ab486a from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:25.041805 9672 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_cd781cda81e34e17b5cc76c5689b9600" } new_table_name: "default.Table_a3dfa51fdb0a42d390d8b0c78c181087" modify_external_catalogs: true
W20250831 01:55:25.042389 10081 leader_election.cc:343] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.042627 10081 leader_election.cc:304] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe14365c42e4c6ba1173d829f997f81; no voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5
W20250831 01:55:25.043150 10081 raft_consensus.cc:2684] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250831 01:55:25.044615 10405 tablet_bootstrap.cc:654] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.044824 10259 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1def9b41fbe94b4ba680d0d4afd0d2b8) from {username='slave'} at 127.0.0.1:48110
I20250831 01:55:25.047504 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1def9b41fbe94b4ba680d0d4afd0d2b8) from {username='slave'} at 127.0.0.1:42118
I20250831 01:55:25.047962 10400 ts_tablet_manager.cc:1918] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.048117 10400 log.cc:1199] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/b61feb48c6164268a2d0b7c892ab486a
I20250831 01:55:25.048663 10400 ts_tablet_manager.cc:1939] T b61feb48c6164268a2d0b7c892ab486a P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:25.050287 9673 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_18f5062db5284a5d9fa9ecc471dc9af7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:25.053880 10397 raft_consensus.cc:491] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.054095 10400 tablet_replica.cc:331] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:25.054126 10397 raft_consensus.cc:513] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.054365 10400 raft_consensus.cc:2241] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.054539 10400 raft_consensus.cc:2270] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.054992 10397 leader_election.cc:290] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.055315 9661 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet b61feb48c6164268a2d0b7c892ab486a (table default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]) successfully deleted
I20250831 01:55:25.055927 10400 ts_tablet_manager.cc:1905] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.056777 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ea7d331b6ee4ff1afaf9ceb8491e5b0" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
W20250831 01:55:25.057446 10080 leader_election.cc:343] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.065927 10400 ts_tablet_manager.cc:1918] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.066120 10400 log.cc:1199] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/7080923b09554076b43599fe5504cfb8
I20250831 01:55:25.066567 10400 ts_tablet_manager.cc:1939] T 7080923b09554076b43599fe5504cfb8 P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:25.068357 9661 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet 7080923b09554076b43599fe5504cfb8 (table default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]) successfully deleted
I20250831 01:55:25.072296 9986 tablet_service.cc:1468] Processing CreateTablet for tablet fdf49ea1050b4326b1df977f021d3d31 (DEFAULT_TABLE table=default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.072821 9986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf49ea1050b4326b1df977f021d3d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.072963 10259 tablet_service.cc:1468] Processing CreateTablet for tablet 1ba0bc1eadc64d219b49437241867a85 (DEFAULT_TABLE table=default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.074257 10125 tablet_service.cc:1468] Processing CreateTablet for tablet 1ba0bc1eadc64d219b49437241867a85 (DEFAULT_TABLE table=default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.074733 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba0bc1eadc64d219b49437241867a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.078809 9670 catalog_manager.cc:2149] created HMS catalog entry for table [id=3e26e3f8edc04fa7a7ed3614967fb1aa]
I20250831 01:55:25.078976 10404 tablet_bootstrap.cc:492] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
W20250831 01:55:25.083153 9662 catalog_manager.cc:4908] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): delete failed for tablet b61feb48c6164268a2d0b7c892ab486a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61feb48c6164268a2d0b7c892ab486a already in progress: creating tablet
W20250831 01:55:25.083490 9662 catalog_manager.cc:4908] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): delete failed for tablet 7080923b09554076b43599fe5504cfb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7080923b09554076b43599fe5504cfb8 already in progress: creating tablet
I20250831 01:55:25.083920 10260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf49ea1050b4326b1df977f021d3d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.084182 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba0bc1eadc64d219b49437241867a85. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 01:55:25.090675 9662 catalog_manager.cc:4908] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): delete failed for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 already in progress: creating tablet
I20250831 01:55:25.090723 10400 tablet_replica.cc:331] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:25.090936 10400 raft_consensus.cc:2241] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.091238 10400 raft_consensus.cc:2270] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.092065 10400 ts_tablet_manager.cc:1905] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.093317 9985 tablet_service.cc:1515] Processing DeleteTablet for tablet 7080923b09554076b43599fe5504cfb8 with delete_type TABLET_DATA_DELETED (replaced by tablet fdf49ea1050b4326b1df977f021d3d31) from {username='slave'} at 127.0.0.1:38376
I20250831 01:55:25.093219 9670 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_b436ecb230034afea5e7f8d551228511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:25.094383 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ea7d331b6ee4ff1afaf9ceb8491e5b0" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
W20250831 01:55:25.095100 10081 leader_election.cc:343] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.095293 10081 leader_election.cc:304] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe14365c42e4c6ba1173d829f997f81; no voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5
W20250831 01:55:25.095589 10081 raft_consensus.cc:2684] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250831 01:55:25.089895 9983 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1def9b41fbe94b4ba680d0d4afd0d2b8) from {username='slave'} at 127.0.0.1:38376
I20250831 01:55:25.098985 10405 log.cc:826] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:25.099331 10400 ts_tablet_manager.cc:1918] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.099484 10400 log.cc:1199] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/0ea7d331b6ee4ff1afaf9ceb8491e5b0
I20250831 01:55:25.099913 10400 ts_tablet_manager.cc:1939] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:25.100095 9987 tablet_service.cc:1468] Processing CreateTablet for tablet 1ba0bc1eadc64d219b49437241867a85 (DEFAULT_TABLE table=default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.100581 9987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba0bc1eadc64d219b49437241867a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.101616 9661 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 (table default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]) successfully deleted
W20250831 01:55:25.103647 9663 catalog_manager.cc:4908] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): delete failed for tablet b61feb48c6164268a2d0b7c892ab486a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61feb48c6164268a2d0b7c892ab486a already in progress: creating tablet
W20250831 01:55:25.104185 9663 catalog_manager.cc:4908] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): delete failed for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 already in progress: creating tablet
W20250831 01:55:25.104475 9663 catalog_manager.cc:4908] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): delete failed for tablet 7080923b09554076b43599fe5504cfb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7080923b09554076b43599fe5504cfb8 already in progress: creating tablet
I20250831 01:55:25.106352 10405 tablet_bootstrap.cc:492] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.106797 10405 ts_tablet_manager.cc:1397] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.000s
I20250831 01:55:25.112098 10405 raft_consensus.cc:357] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.112488 10405 raft_consensus.cc:383] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.112598 10405 raft_consensus.cc:738] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.113435 10405 consensus_queue.cc:260] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.116173 10259 tablet_service.cc:1468] Processing CreateTablet for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 (DEFAULT_TABLE table=default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.116727 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.126360 9984 tablet_service.cc:1468] Processing CreateTablet for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 (DEFAULT_TABLE table=default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.127112 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.131192 10397 raft_consensus.cc:491] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.131440 10397 raft_consensus.cc:513] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.132440 10397 leader_election.cc:290] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.133661 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe4c7b8517747a9a9baef5c25d314f6" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.134076 10008 raft_consensus.cc:2466] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.134825 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe4c7b8517747a9a9baef5c25d314f6" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.135792 10081 leader_election.cc:304] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
W20250831 01:55:25.136389 10080 leader_election.cc:343] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.136811 10397 raft_consensus.cc:2802] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.136970 10397 raft_consensus.cc:491] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.137086 10397 raft_consensus.cc:3058] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.139369 9671 catalog_manager.cc:2149] created HMS catalog entry for table [id=b8a4751b36dc49a6906d7734ef84069a]
I20250831 01:55:25.141023 10125 tablet_service.cc:1468] Processing CreateTablet for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 (DEFAULT_TABLE table=default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.141605 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.145524 10395 tablet_bootstrap.cc:492] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.145709 10395 ts_tablet_manager.cc:1397] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.116s user 0.029s sys 0.025s
I20250831 01:55:25.146603 10395 raft_consensus.cc:357] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.146703 10404 tablet_bootstrap.cc:654] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.146778 10395 raft_consensus.cc:383] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.146872 10395 raft_consensus.cc:738] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.147123 10395 consensus_queue.cc:260] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.148896 9671 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_e32aefba1e29406f91d7559a02f1323f" } modify_external_catalogs: true
I20250831 01:55:25.149163 9671 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_e32aefba1e29406f91d7559a02f1323f" } modify_external_catalogs: true
I20250831 01:55:25.151667 10395 ts_tablet_manager.cc:1428] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250831 01:55:25.153625 10404 log.cc:826] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:25.153915 10395 tablet_bootstrap.cc:492] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.155726 10395 tablet_bootstrap.cc:654] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.162421 10405 ts_tablet_manager.cc:1428] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.055s user 0.014s sys 0.028s
I20250831 01:55:25.162650 10397 raft_consensus.cc:513] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.164098 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe4c7b8517747a9a9baef5c25d314f6" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.164372 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe4c7b8517747a9a9baef5c25d314f6" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:25.164615 10008 raft_consensus.cc:3058] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
W20250831 01:55:25.164741 10080 leader_election.cc:343] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.164989 10397 leader_election.cc:290] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.166709 10259 tablet_service.cc:1468] Processing CreateTablet for tablet d36d4612593e4d8985b160918d54f87a (DEFAULT_TABLE table=default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.167129 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36d4612593e4d8985b160918d54f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.167404 10404 tablet_bootstrap.cc:492] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.167577 10008 raft_consensus.cc:2466] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.167748 10404 ts_tablet_manager.cc:1397] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.004s
I20250831 01:55:25.168314 10081 leader_election.cc:304] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters: 903e5d68fe5e4988a6b54623da741af5
I20250831 01:55:25.169191 10397 raft_consensus.cc:2802] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.169368 10397 raft_consensus.cc:695] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:25.169829 10397 consensus_queue.cc:237] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.172274 9984 tablet_service.cc:1468] Processing CreateTablet for tablet d36d4612593e4d8985b160918d54f87a (DEFAULT_TABLE table=default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.172731 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36d4612593e4d8985b160918d54f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.176709 10125 tablet_service.cc:1468] Processing CreateTablet for tablet d36d4612593e4d8985b160918d54f87a (DEFAULT_TABLE table=default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.177122 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36d4612593e4d8985b160918d54f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.179001 10395 tablet_bootstrap.cc:492] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.179189 10395 ts_tablet_manager.cc:1397] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.002s
I20250831 01:55:25.179991 10395 raft_consensus.cc:357] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.180182 10395 raft_consensus.cc:383] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.180255 10395 raft_consensus.cc:738] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.180517 10395 consensus_queue.cc:260] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.181031 10395 ts_tablet_manager.cc:1428] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250831 01:55:25.181340 10395 tablet_bootstrap.cc:492] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.183111 10395 tablet_bootstrap.cc:654] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.187489 10405 tablet_bootstrap.cc:492] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.189872 10405 tablet_bootstrap.cc:654] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.196609 9984 tablet_service.cc:1468] Processing CreateTablet for tablet d596b92ea9d745098c779e349a551e82 (DEFAULT_TABLE table=default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.197160 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d596b92ea9d745098c779e349a551e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.199321 10404 raft_consensus.cc:357] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.199502 10259 tablet_service.cc:1468] Processing CreateTablet for tablet d596b92ea9d745098c779e349a551e82 (DEFAULT_TABLE table=default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.200089 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d596b92ea9d745098c779e349a551e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.204151 10125 tablet_service.cc:1468] Processing CreateTablet for tablet d596b92ea9d745098c779e349a551e82 (DEFAULT_TABLE table=default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.204648 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d596b92ea9d745098c779e349a551e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.199698 10404 raft_consensus.cc:383] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.205472 10404 raft_consensus.cc:738] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.206789 10404 consensus_queue.cc:260] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.214673 10404 ts_tablet_manager.cc:1428] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.047s user 0.008s sys 0.003s
I20250831 01:55:25.218559 10405 tablet_bootstrap.cc:492] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.218740 10405 ts_tablet_manager.cc:1397] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.003s
I20250831 01:55:25.219208 10395 tablet_bootstrap.cc:492] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.219358 10395 ts_tablet_manager.cc:1397] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.005s
I20250831 01:55:25.219857 10405 raft_consensus.cc:357] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.220052 10405 raft_consensus.cc:383] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.220162 10405 raft_consensus.cc:738] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.220481 10405 consensus_queue.cc:260] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.220852 10395 raft_consensus.cc:357] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.221030 10395 raft_consensus.cc:383] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.221118 10395 raft_consensus.cc:738] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.221362 10395 consensus_queue.cc:260] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.221964 10395 ts_tablet_manager.cc:1428] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250831 01:55:25.223106 10405 ts_tablet_manager.cc:1428] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250831 01:55:25.224097 10395 tablet_bootstrap.cc:492] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.226030 10395 tablet_bootstrap.cc:654] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.228489 10405 tablet_bootstrap.cc:492] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.230085 10404 tablet_bootstrap.cc:492] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.231736 10404 tablet_bootstrap.cc:654] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.232965 10405 tablet_bootstrap.cc:654] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.234691 10404 tablet_bootstrap.cc:492] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.234836 10404 ts_tablet_manager.cc:1397] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250831 01:55:25.235756 10404 raft_consensus.cc:357] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.235951 10404 raft_consensus.cc:383] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.236032 10404 raft_consensus.cc:738] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.239195 10395 tablet_bootstrap.cc:492] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.239392 10395 ts_tablet_manager.cc:1397] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
W20250831 01:55:25.239455 10062 tablet.cc:2378] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 01:55:25.240228 10395 raft_consensus.cc:357] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.240507 10395 raft_consensus.cc:383] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.240615 10395 raft_consensus.cc:738] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.240793 10404 consensus_queue.cc:260] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.240890 10395 consensus_queue.cc:260] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.244163 9669 catalog_manager.cc:2149] created HMS catalog entry for table [id=6b6b5b6d016b43f0921a4b22fe2aadd5]
I20250831 01:55:25.245019 10404 ts_tablet_manager.cc:1428] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250831 01:55:25.245066 10395 ts_tablet_manager.cc:1428] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250831 01:55:25.248579 10395 tablet_bootstrap.cc:492] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.250564 10395 tablet_bootstrap.cc:654] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.251107 10404 tablet_bootstrap.cc:492] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.252089 10405 tablet_bootstrap.cc:492] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.252291 10405 ts_tablet_manager.cc:1397] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250831 01:55:25.253154 10405 raft_consensus.cc:357] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.253412 10405 raft_consensus.cc:383] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.253506 10405 raft_consensus.cc:738] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.253751 10405 consensus_queue.cc:260] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.255856 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--webserver_interface=127.9.47.125
--webserver_port=46501
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250831 01:55:25.256875 10405 ts_tablet_manager.cc:1428] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250831 01:55:25.260064 9669 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_71f3e7f55e21496389aaf63260dec6a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:25.260396 10405 tablet_bootstrap.cc:492] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.266112 10404 tablet_bootstrap.cc:654] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.267638 10405 tablet_bootstrap.cc:654] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.276369 10395 tablet_bootstrap.cc:492] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.276532 10404 tablet_bootstrap.cc:492] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.276577 10395 ts_tablet_manager.cc:1397] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.013s
I20250831 01:55:25.276712 10404 ts_tablet_manager.cc:1397] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.002s
I20250831 01:55:25.277082 10125 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee86091bc604b6a9d33c56d091fbd4d (DEFAULT_TABLE table=default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.277508 10395 raft_consensus.cc:357] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.277503 10404 raft_consensus.cc:357] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.277627 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee86091bc604b6a9d33c56d091fbd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.277727 10404 raft_consensus.cc:383] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.278409 9984 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee86091bc604b6a9d33c56d091fbd4d (DEFAULT_TABLE table=default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.278857 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee86091bc604b6a9d33c56d091fbd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.279170 10259 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee86091bc604b6a9d33c56d091fbd4d (DEFAULT_TABLE table=default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.279606 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee86091bc604b6a9d33c56d091fbd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.284255 10395 raft_consensus.cc:383] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.284402 10395 raft_consensus.cc:738] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.284694 10395 consensus_queue.cc:260] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.286207 10404 raft_consensus.cc:738] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.286525 10404 consensus_queue.cc:260] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.287030 10395 ts_tablet_manager.cc:1428] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250831 01:55:25.288197 10395 tablet_bootstrap.cc:492] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.288568 10404 ts_tablet_manager.cc:1428] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250831 01:55:25.289785 10404 tablet_bootstrap.cc:492] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.291168 10395 tablet_bootstrap.cc:654] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.291489 10404 tablet_bootstrap.cc:654] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.293438 10405 tablet_bootstrap.cc:492] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.293685 10405 ts_tablet_manager.cc:1397] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.005s
I20250831 01:55:25.294798 10405 raft_consensus.cc:357] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.295085 10405 raft_consensus.cc:383] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.295205 10405 raft_consensus.cc:738] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.295499 10405 consensus_queue.cc:260] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.300318 10405 ts_tablet_manager.cc:1428] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250831 01:55:25.304409 10405 tablet_bootstrap.cc:492] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.306835 10405 tablet_bootstrap.cc:654] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.309538 10395 tablet_bootstrap.cc:492] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.309697 10404 tablet_bootstrap.cc:492] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.309733 10395 ts_tablet_manager.cc:1397] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20250831 01:55:25.309875 10404 ts_tablet_manager.cc:1397] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250831 01:55:25.310570 10395 raft_consensus.cc:357] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.310678 10404 raft_consensus.cc:357] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.310792 10395 raft_consensus.cc:383] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.310864 10395 raft_consensus.cc:738] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.311187 10395 consensus_queue.cc:260] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.310870 10404 raft_consensus.cc:383] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.311475 10404 raft_consensus.cc:738] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.311738 10404 consensus_queue.cc:260] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.313158 10395 ts_tablet_manager.cc:1428] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250831 01:55:25.314421 10395 tablet_bootstrap.cc:492] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.314556 10404 ts_tablet_manager.cc:1428] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250831 01:55:25.315675 10404 tablet_bootstrap.cc:492] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.316265 10395 tablet_bootstrap.cc:654] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.320055 10404 tablet_bootstrap.cc:654] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.322012 9668 catalog_manager.cc:2149] created HMS catalog entry for table [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]
I20250831 01:55:25.322227 10405 tablet_bootstrap.cc:492] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.322491 10405 ts_tablet_manager.cc:1397] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250831 01:55:25.323467 10405 raft_consensus.cc:357] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.323694 10405 raft_consensus.cc:383] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.323792 10405 raft_consensus.cc:738] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.324246 10395 tablet_bootstrap.cc:492] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.324134 10405 consensus_queue.cc:260] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.324398 10395 ts_tablet_manager.cc:1397] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250831 01:55:25.325191 10395 raft_consensus.cc:357] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.325383 10395 raft_consensus.cc:383] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.325490 10395 raft_consensus.cc:738] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.325789 10395 consensus_queue.cc:260] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.327337 10395 ts_tablet_manager.cc:1428] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250831 01:55:25.327885 10405 ts_tablet_manager.cc:1428] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250831 01:55:25.328428 10395 tablet_bootstrap.cc:492] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.330955 10405 tablet_bootstrap.cc:492] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.332695 10405 tablet_bootstrap.cc:654] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.332998 9668 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 10) took 1053 ms. Trace:
I20250831 01:55:25.334204 10395 tablet_bootstrap.cc:654] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.333149 9668 rpcz_store.cc:276] 0831 01:55:24.279662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:24.279747 (+ 85us) service_pool.cc:225] Handling call
0831 01:55:25.037639 (+757892us) catalog_manager.cc:2076] Acquired catalog manager lock
0831 01:55:25.038136 (+ 497us) catalog_manager.cc:2122] Created new table and tablet info
0831 01:55:25.321959 (+283823us) catalog_manager.cc:2148] Created new table in HMS catalog
0831 01:55:25.332696 (+ 10737us) catalog_manager.cc:2182] Wrote table and tablets to system table
0831 01:55:25.332835 (+ 139us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0831 01:55:25.332945 (+ 110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 01:55:25.322894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 01:55:25.323017 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0831 01:55:25.323024 (+ 7us) write_op.cc:435] Acquired schema lock
0831 01:55:25.323028 (+ 4us) tablet.cc:660] Decoding operations
0831 01:55:25.323124 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0831 01:55:25.323138 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0831 01:55:25.323142 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0831 01:55:25.323223 (+ 81us) tablet.cc:699] Row locks acquired
0831 01:55:25.323226 (+ 3us) write_op.cc:260] PREPARE: finished
0831 01:55:25.323267 (+ 41us) write_op.cc:270] Start()
0831 01:55:25.323304 (+ 37us) write_op.cc:276] Timestamp: P: 1756605325323235 usec, L: 0
0831 01:55:25.323307 (+ 3us) op_driver.cc:348] REPLICATION: starting
0831 01:55:25.323495 (+ 188us) log.cc:844] Serialized 432 byte log entry
0831 01:55:25.331274 (+ 7779us) op_driver.cc:464] REPLICATION: finished
0831 01:55:25.331448 (+ 174us) write_op.cc:301] APPLY: starting
0831 01:55:25.331466 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0831 01:55:25.331515 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0831 01:55:25.331552 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 01:55:25.331664 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 01:55:25.331701 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0831 01:55:25.331708 (+ 7us) write_op.cc:312] APPLY: finished
0831 01:55:25.331793 (+ 85us) log.cc:844] Serialized 28 byte log entry
0831 01:55:25.331862 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:25.331884 (+ 22us) write_op.cc:454] Released schema lock
0831 01:55:25.331891 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0831 01:55:25.331966 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:25.331969 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":205648,"Hive Metastore.run_cpu_time_us":286,"Hive Metastore.run_wall_time_us":77728,"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":511,"apply.run_wall_time_us":511,"num_ops":2,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":727,"raft.queue_time_us":1171,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":276,"replication_time_us":7932}]]}
I20250831 01:55:25.334888 10404 tablet_bootstrap.cc:492] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.335068 10404 ts_tablet_manager.cc:1397] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250831 01:55:25.335923 10404 raft_consensus.cc:357] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.336174 10404 raft_consensus.cc:383] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.336272 10404 raft_consensus.cc:738] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.336599 10404 consensus_queue.cc:260] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.338549 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1def9b41fbe94b4ba680d0d4afd0d2b8) from {username='slave'} at 127.0.0.1:38376
I20250831 01:55:25.338923 10403 tablet_replica.cc:331] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:25.339516 10403 raft_consensus.cc:2241] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.339984 10403 raft_consensus.cc:2270] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.341897 10403 ts_tablet_manager.cc:1905] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.342211 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet b61feb48c6164268a2d0b7c892ab486a with delete_type TABLET_DATA_DELETED (replaced by tablet 3772a8213fd74d1a995e0c8eed42c08b) from {username='slave'} at 127.0.0.1:38376
I20250831 01:55:25.343655 10404 ts_tablet_manager.cc:1428] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250831 01:55:25.348739 10403 ts_tablet_manager.cc:1918] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.349018 10404 tablet_bootstrap.cc:492] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.349359 10125 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd06ceda8134e459d5bb86fac5cdbb8 (DEFAULT_TABLE table=default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.349857 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd06ceda8134e459d5bb86fac5cdbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.350868 10403 log.cc:1199] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/0ea7d331b6ee4ff1afaf9ceb8491e5b0
I20250831 01:55:25.351389 10403 ts_tablet_manager.cc:1939] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:25.351660 10404 tablet_bootstrap.cc:654] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.348770 9984 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd06ceda8134e459d5bb86fac5cdbb8 (DEFAULT_TABLE table=default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.353826 10405 tablet_bootstrap.cc:492] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.354012 10405 ts_tablet_manager.cc:1397] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20250831 01:55:25.354130 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd06ceda8134e459d5bb86fac5cdbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.355094 10405 raft_consensus.cc:357] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.355366 10405 raft_consensus.cc:383] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.355491 10405 raft_consensus.cc:738] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.355787 10405 consensus_queue.cc:260] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.357352 10259 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd06ceda8134e459d5bb86fac5cdbb8 (DEFAULT_TABLE table=default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:25.357793 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd06ceda8134e459d5bb86fac5cdbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:25.360960 10405 ts_tablet_manager.cc:1428] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250831 01:55:25.362820 10405 tablet_bootstrap.cc:492] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.365065 10405 tablet_bootstrap.cc:654] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.367327 9668 catalog_manager.cc:5582] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66). New cstate: current_term: 1 leader_uuid: "fbe14365c42e4c6ba1173d829f997f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } }
I20250831 01:55:25.369732 9663 catalog_manager.cc:4928] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 (table default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]) successfully deleted
I20250831 01:55:25.370151 10403 tablet_replica.cc:331] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:25.370424 10403 raft_consensus.cc:2241] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.370606 10403 raft_consensus.cc:2270] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.371943 10403 ts_tablet_manager.cc:1905] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.373576 10404 tablet_bootstrap.cc:492] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.373800 10404 ts_tablet_manager.cc:1397] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20250831 01:55:25.374864 10404 raft_consensus.cc:357] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.375111 10404 raft_consensus.cc:383] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.375212 10404 raft_consensus.cc:738] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.378914 10395 tablet_bootstrap.cc:492] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.379137 10395 ts_tablet_manager.cc:1397] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.051s user 0.025s sys 0.015s
I20250831 01:55:25.380192 10395 raft_consensus.cc:357] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.380419 10395 raft_consensus.cc:383] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.380508 10395 raft_consensus.cc:738] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.380775 10395 consensus_queue.cc:260] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.382290 10403 ts_tablet_manager.cc:1918] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.382419 10395 ts_tablet_manager.cc:1428] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250831 01:55:25.382440 10403 log.cc:1199] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/b61feb48c6164268a2d0b7c892ab486a
I20250831 01:55:25.382798 10403 ts_tablet_manager.cc:1939] T b61feb48c6164268a2d0b7c892ab486a P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:25.383627 10395 tablet_bootstrap.cc:492] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:25.383865 9663 catalog_manager.cc:4928] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): tablet b61feb48c6164268a2d0b7c892ab486a (table default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]) successfully deleted
I20250831 01:55:25.384213 10404 consensus_queue.cc:260] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.385371 10395 tablet_bootstrap.cc:654] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.386750 10259 tablet_service.cc:1515] Processing DeleteTablet for tablet b61feb48c6164268a2d0b7c892ab486a with delete_type TABLET_DATA_DELETED (replaced by tablet 3772a8213fd74d1a995e0c8eed42c08b) from {username='slave'} at 127.0.0.1:48110
I20250831 01:55:25.387116 10401 tablet_replica.cc:331] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:25.387606 10401 raft_consensus.cc:2241] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.387943 10404 ts_tablet_manager.cc:1428] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250831 01:55:25.388026 10401 raft_consensus.cc:2270] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.390918 10401 ts_tablet_manager.cc:1905] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.401403 10401 ts_tablet_manager.cc:1918] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.403323 10405 tablet_bootstrap.cc:492] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.403667 10405 ts_tablet_manager.cc:1397] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.003s
I20250831 01:55:25.404856 10405 raft_consensus.cc:357] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.405236 10405 raft_consensus.cc:383] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.405400 10405 raft_consensus.cc:738] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.405877 10405 consensus_queue.cc:260] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.410311 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet 7080923b09554076b43599fe5504cfb8 with delete_type TABLET_DATA_DELETED (replaced by tablet fdf49ea1050b4326b1df977f021d3d31) from {username='slave'} at 127.0.0.1:38376
I20250831 01:55:25.410887 10405 ts_tablet_manager.cc:1428] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250831 01:55:25.411140 10401 log.cc:1199] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/b61feb48c6164268a2d0b7c892ab486a
I20250831 01:55:25.411418 10395 tablet_bootstrap.cc:492] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:25.402021 10404 tablet_bootstrap.cc:492] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.411607 10395 ts_tablet_manager.cc:1397] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.013s
I20250831 01:55:25.412475 10395 raft_consensus.cc:357] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.412691 10395 raft_consensus.cc:383] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.412782 10395 raft_consensus.cc:738] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.415144 10395 consensus_queue.cc:260] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.414580 10405 tablet_bootstrap.cc:492] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.413290 10401 ts_tablet_manager.cc:1939] T b61feb48c6164268a2d0b7c892ab486a P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:25.416795 10395 ts_tablet_manager.cc:1428] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250831 01:55:25.417333 10404 tablet_bootstrap.cc:654] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.419302 10405 tablet_bootstrap.cc:654] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.420967 10403 tablet_replica.cc:331] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:25.420979 9662 catalog_manager.cc:4928] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): tablet b61feb48c6164268a2d0b7c892ab486a (table default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194]) successfully deleted
I20250831 01:55:25.421159 10403 raft_consensus.cc:2241] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.421365 10403 raft_consensus.cc:2270] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.422112 10404 tablet_bootstrap.cc:492] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.422283 10404 ts_tablet_manager.cc:1397] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250831 01:55:25.423172 10259 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1def9b41fbe94b4ba680d0d4afd0d2b8) from {username='slave'} at 127.0.0.1:48110
I20250831 01:55:25.423131 10404 raft_consensus.cc:357] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.423352 10404 raft_consensus.cc:383] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.423440 10404 raft_consensus.cc:738] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.423416 10258 tablet_service.cc:1515] Processing DeleteTablet for tablet 7080923b09554076b43599fe5504cfb8 with delete_type TABLET_DATA_DELETED (replaced by tablet fdf49ea1050b4326b1df977f021d3d31) from {username='slave'} at 127.0.0.1:48110
I20250831 01:55:25.423672 10401 tablet_replica.cc:331] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:25.423877 10401 raft_consensus.cc:2241] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.424018 10401 raft_consensus.cc:2270] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.424343 10404 consensus_queue.cc:260] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.424883 10401 ts_tablet_manager.cc:1905] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.425935 10397 raft_consensus.cc:491] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.426102 10397 raft_consensus.cc:513] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.426725 10397 leader_election.cc:290] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.427387 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d596b92ea9d745098c779e349a551e82" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
W20250831 01:55:25.427983 10081 leader_election.cc:343] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.428736 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d596b92ea9d745098c779e349a551e82" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
W20250831 01:55:25.429349 10080 leader_election.cc:343] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.429466 10080 leader_election.cc:304] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe14365c42e4c6ba1173d829f997f81; no voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5
I20250831 01:55:25.429982 10397 raft_consensus.cc:2747] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 01:55:25.430518 10403 ts_tablet_manager.cc:1905] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.431912 9668 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_e5d7d8712d2d4ec181665ef359124956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:25.435772 10404 ts_tablet_manager.cc:1428] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250831 01:55:25.437124 10404 tablet_bootstrap.cc:492] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.439381 10405 tablet_bootstrap.cc:492] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.439538 10405 ts_tablet_manager.cc:1397] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20250831 01:55:25.440193 10405 raft_consensus.cc:357] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.440354 10405 raft_consensus.cc:383] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.440443 10405 raft_consensus.cc:738] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.440672 10405 consensus_queue.cc:260] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.442282 10405 ts_tablet_manager.cc:1428] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250831 01:55:25.443393 10405 tablet_bootstrap.cc:492] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
W20250831 01:55:25.444561 10328 tablet.cc:2378] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 01:55:25.446039 10404 tablet_bootstrap.cc:654] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.446327 10405 tablet_bootstrap.cc:654] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.448599 10401 ts_tablet_manager.cc:1918] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.448747 10401 log.cc:1199] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/0ea7d331b6ee4ff1afaf9ceb8491e5b0
I20250831 01:55:25.449113 10401 ts_tablet_manager.cc:1939] T 0ea7d331b6ee4ff1afaf9ceb8491e5b0 P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:25.450294 9662 catalog_manager.cc:4928] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): tablet 0ea7d331b6ee4ff1afaf9ceb8491e5b0 (table default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b]) successfully deleted
I20250831 01:55:25.450696 10401 tablet_replica.cc:331] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:25.450951 10401 raft_consensus.cc:2241] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:25.451153 10401 raft_consensus.cc:2270] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:25.451191 10403 ts_tablet_manager.cc:1918] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.451613 10403 log.cc:1199] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/7080923b09554076b43599fe5504cfb8
I20250831 01:55:25.452144 10401 ts_tablet_manager.cc:1905] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:25.452596 10403 ts_tablet_manager.cc:1939] T 7080923b09554076b43599fe5504cfb8 P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:25.454556 10404 tablet_bootstrap.cc:492] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.454739 10404 ts_tablet_manager.cc:1397] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250831 01:55:25.455641 10404 raft_consensus.cc:357] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.455837 10404 raft_consensus.cc:383] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.455929 10404 raft_consensus.cc:738] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.455912 9663 catalog_manager.cc:4928] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): tablet 7080923b09554076b43599fe5504cfb8 (table default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]) successfully deleted
I20250831 01:55:25.456166 10404 consensus_queue.cc:260] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.456732 10404 ts_tablet_manager.cc:1428] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250831 01:55:25.457747 10405 tablet_bootstrap.cc:492] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.457970 10405 ts_tablet_manager.cc:1397] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250831 01:55:25.458806 10405 raft_consensus.cc:357] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.459018 10405 raft_consensus.cc:383] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.459128 10405 raft_consensus.cc:738] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.459405 10405 consensus_queue.cc:260] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.460070 10405 ts_tablet_manager.cc:1428] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250831 01:55:25.464385 10404 tablet_bootstrap.cc:492] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.465054 10405 tablet_bootstrap.cc:492] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.467087 10405 tablet_bootstrap.cc:654] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.471515 10401 ts_tablet_manager.cc:1918] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250831 01:55:25.471985 10404 tablet_bootstrap.cc:654] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.472975 10401 log.cc:1199] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/7080923b09554076b43599fe5504cfb8
I20250831 01:55:25.473438 10401 ts_tablet_manager.cc:1939] T 7080923b09554076b43599fe5504cfb8 P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:25.474732 9662 catalog_manager.cc:4928] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): tablet 7080923b09554076b43599fe5504cfb8 (table default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178]) successfully deleted
I20250831 01:55:25.475226 10405 tablet_bootstrap.cc:492] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.475436 10405 ts_tablet_manager.cc:1397] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20250831 01:55:25.476342 10405 raft_consensus.cc:357] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.476547 10405 raft_consensus.cc:383] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.476648 10405 raft_consensus.cc:738] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.476910 10405 consensus_queue.cc:260] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.479872 10405 ts_tablet_manager.cc:1428] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250831 01:55:25.482447 10405 tablet_bootstrap.cc:492] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.484304 10405 tablet_bootstrap.cc:654] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.486439 10404 tablet_bootstrap.cc:492] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.486644 10404 ts_tablet_manager.cc:1397] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20250831 01:55:25.487429 10404 raft_consensus.cc:357] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.487644 10404 raft_consensus.cc:383] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.487720 10404 raft_consensus.cc:738] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.487978 10404 consensus_queue.cc:260] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.489921 10404 ts_tablet_manager.cc:1428] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250831 01:55:25.491734 10404 tablet_bootstrap.cc:492] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.493541 10404 tablet_bootstrap.cc:654] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.495471 10405 tablet_bootstrap.cc:492] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.495664 10405 ts_tablet_manager.cc:1397] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20250831 01:55:25.495718 10397 raft_consensus.cc:491] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.495910 10397 raft_consensus.cc:513] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.496512 10405 raft_consensus.cc:357] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.496636 10397 leader_election.cc:290] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.496750 10405 raft_consensus.cc:383] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.496835 10405 raft_consensus.cc:738] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.497191 10405 consensus_queue.cc:260] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.497428 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdf49ea1050b4326b1df977f021d3d31" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.497741 10280 raft_consensus.cc:2466] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.497877 10405 ts_tablet_manager.cc:1428] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250831 01:55:25.500303 10405 tablet_bootstrap.cc:492] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.500774 10080 leader_election.cc:304] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.502024 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdf49ea1050b4326b1df977f021d3d31" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.502149 10397 raft_consensus.cc:2802] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.502246 10397 raft_consensus.cc:491] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.502261 10008 raft_consensus.cc:2466] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.502351 10397 raft_consensus.cc:3058] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.502010 10405 tablet_bootstrap.cc:654] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.505101 10397 raft_consensus.cc:513] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.506518 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdf49ea1050b4326b1df977f021d3d31" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:25.506742 10008 raft_consensus.cc:3058] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.509548 10008 raft_consensus.cc:2466] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.510185 10081 leader_election.cc:304] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.510577 10411 raft_consensus.cc:2802] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.516021 10405 tablet_bootstrap.cc:492] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.516206 10405 ts_tablet_manager.cc:1397] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20250831 01:55:25.517026 10405 raft_consensus.cc:357] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.517220 10405 raft_consensus.cc:383] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.517307 10405 raft_consensus.cc:738] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.517575 10405 consensus_queue.cc:260] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.518232 10405 ts_tablet_manager.cc:1428] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250831 01:55:25.518538 10405 tablet_bootstrap.cc:492] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:25.520335 10405 tablet_bootstrap.cc:654] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.526808 10397 leader_election.cc:290] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.527774 10397 raft_consensus.cc:491] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.528040 10397 raft_consensus.cc:513] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.529521 10397 leader_election.cc:290] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.529860 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba0bc1eadc64d219b49437241867a85" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.530135 10008 raft_consensus.cc:2466] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.530669 10081 leader_election.cc:304] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.534970 10397 raft_consensus.cc:2802] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.535115 10397 raft_consensus.cc:491] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.535275 10397 raft_consensus.cc:3058] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.538321 10397 raft_consensus.cc:513] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.541328 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba0bc1eadc64d219b49437241867a85" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:25.541620 10008 raft_consensus.cc:3058] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.544751 10008 raft_consensus.cc:2466] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.545329 10081 leader_election.cc:304] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.550771 10404 tablet_bootstrap.cc:492] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.550958 10404 ts_tablet_manager.cc:1397] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.003s
I20250831 01:55:25.551793 10404 raft_consensus.cc:357] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.551986 10404 raft_consensus.cc:383] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.552074 10404 raft_consensus.cc:738] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.552309 10404 consensus_queue.cc:260] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.552785 10404 ts_tablet_manager.cc:1428] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250831 01:55:25.554787 10404 tablet_bootstrap.cc:492] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.556491 10404 tablet_bootstrap.cc:654] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.563879 10397 leader_election.cc:290] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.564258 10397 raft_consensus.cc:2802] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.564515 10397 raft_consensus.cc:695] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:25.564922 10397 consensus_queue.cc:237] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.568346 10411 raft_consensus.cc:695] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:25.569726 10411 consensus_queue.cc:237] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.571906 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdf49ea1050b4326b1df977f021d3d31" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.572124 10280 raft_consensus.cc:3058] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.575377 10280 raft_consensus.cc:2466] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.576349 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba0bc1eadc64d219b49437241867a85" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.576615 10280 raft_consensus.cc:2466] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.577476 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba0bc1eadc64d219b49437241867a85" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.577711 10280 raft_consensus.cc:3058] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.580680 10280 raft_consensus.cc:2466] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.590833 10405 tablet_bootstrap.cc:492] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:25.591145 10405 ts_tablet_manager.cc:1397] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.073s user 0.024s sys 0.042s
I20250831 01:55:25.591429 10404 tablet_bootstrap.cc:492] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.591629 10404 ts_tablet_manager.cc:1397] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.007s
I20250831 01:55:25.592193 10405 raft_consensus.cc:357] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.592350 10404 raft_consensus.cc:357] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.592496 10405 raft_consensus.cc:383] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.592597 10404 raft_consensus.cc:383] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.592679 10404 raft_consensus.cc:738] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.592705 10405 raft_consensus.cc:738] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.593005 10404 consensus_queue.cc:260] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.593120 10405 consensus_queue.cc:260] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.593744 10405 ts_tablet_manager.cc:1428] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250831 01:55:25.597327 10397 raft_consensus.cc:491] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.597571 10397 raft_consensus.cc:513] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.598338 10397 leader_election.cc:290] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.598951 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee86091bc604b6a9d33c56d091fbd4d" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.599324 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee86091bc604b6a9d33c56d091fbd4d" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
W20250831 01:55:25.599517 10080 leader_election.cc:343] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.599550 10008 raft_consensus.cc:2466] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.600118 10081 leader_election.cc:304] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters: 903e5d68fe5e4988a6b54623da741af5
I20250831 01:55:25.600931 10397 raft_consensus.cc:2802] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.601048 10397 raft_consensus.cc:491] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.601143 10397 raft_consensus.cc:3058] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.603879 10397 raft_consensus.cc:513] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.604980 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee86091bc604b6a9d33c56d091fbd4d" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:25.605201 10008 raft_consensus.cc:3058] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.608088 10008 raft_consensus.cc:2466] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.608644 10081 leader_election.cc:304] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.608963 10411 raft_consensus.cc:2802] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.609122 10411 raft_consensus.cc:695] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:25.609422 10411 consensus_queue.cc:237] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.614632 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee86091bc604b6a9d33c56d091fbd4d" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
W20250831 01:55:25.615397 10080 leader_election.cc:343] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:25.615633 10445 raft_consensus.cc:491] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.615796 10445 raft_consensus.cc:513] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.616658 10445 leader_election.cc:290] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.617388 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbcda68336141109d5e3c96f97454a5" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.617647 10280 raft_consensus.cc:2466] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.618161 10080 leader_election.cc:304] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.618485 10445 raft_consensus.cc:2802] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.618606 10445 raft_consensus.cc:491] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.618693 10445 raft_consensus.cc:3058] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.621455 10445 raft_consensus.cc:513] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.622387 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbcda68336141109d5e3c96f97454a5" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.622627 10008 raft_consensus.cc:2466] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.624082 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbcda68336141109d5e3c96f97454a5" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.624281 10280 raft_consensus.cc:3058] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.627763 10280 raft_consensus.cc:2466] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.628392 10080 leader_election.cc:304] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.628700 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbcda68336141109d5e3c96f97454a5" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:25.628906 10008 raft_consensus.cc:3058] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.628892 10444 raft_consensus.cc:2802] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.631661 10008 raft_consensus.cc:2466] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.632184 10445 leader_election.cc:290] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.632753 10397 leader_election.cc:290] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.635826 10404 ts_tablet_manager.cc:1428] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.044s user 0.000s sys 0.002s
I20250831 01:55:25.638195 10404 tablet_bootstrap.cc:492] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.639860 10404 tablet_bootstrap.cc:654] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.646225 10444 raft_consensus.cc:695] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:25.646677 10444 consensus_queue.cc:237] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.654721 10451 raft_consensus.cc:491] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.654860 10404 tablet_bootstrap.cc:492] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.655035 10404 ts_tablet_manager.cc:1397] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250831 01:55:25.654960 10451 raft_consensus.cc:513] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.655887 10404 raft_consensus.cc:357] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.656044 10451 leader_election.cc:290] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.656086 10404 raft_consensus.cc:383] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.656185 10404 raft_consensus.cc:738] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.656450 10404 consensus_queue.cc:260] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.656567 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d36d4612593e4d8985b160918d54f87a" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.656606 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d36d4612593e4d8985b160918d54f87a" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.656786 10008 raft_consensus.cc:2466] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.656821 10280 raft_consensus.cc:2466] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.657004 10404 ts_tablet_manager.cc:1428] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250831 01:55:25.657279 10404 tablet_bootstrap.cc:492] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:25.657437 10080 leader_election.cc:304] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.657758 10451 raft_consensus.cc:2802] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.657899 10451 raft_consensus.cc:491] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.657996 10451 raft_consensus.cc:3058] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.659353 10404 tablet_bootstrap.cc:654] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:25.660775 10451 raft_consensus.cc:513] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.661815 10451 leader_election.cc:290] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.661791 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d36d4612593e4d8985b160918d54f87a" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:25.662053 10008 raft_consensus.cc:3058] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.662480 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d36d4612593e4d8985b160918d54f87a" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.662675 10280 raft_consensus.cc:3058] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.664852 10008 raft_consensus.cc:2466] T d36d4612593e4d8985b160918d54f87a P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.665258 10081 leader_election.cc:304] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.665285 10280 raft_consensus.cc:2466] T d36d4612593e4d8985b160918d54f87a P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.665638 10451 raft_consensus.cc:2802] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.666749 10451 raft_consensus.cc:695] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:25.667055 10451 consensus_queue.cc:237] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.675145 10404 tablet_bootstrap.cc:492] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:25.675360 10404 ts_tablet_manager.cc:1397] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20250831 01:55:25.676146 10404 raft_consensus.cc:357] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.676344 10404 raft_consensus.cc:383] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:25.676422 10404 raft_consensus.cc:738] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:25.676712 10404 consensus_queue.cc:260] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.678400 10404 ts_tablet_manager.cc:1428] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250831 01:55:25.694198 10417 raft_consensus.cc:491] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.694430 10417 raft_consensus.cc:513] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.696254 10417 leader_election.cc:290] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:25.721140 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee86091bc604b6a9d33c56d091fbd4d" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:25.723217 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee86091bc604b6a9d33c56d091fbd4d" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.723608 10008 raft_consensus.cc:2391] T 4ee86091bc604b6a9d33c56d091fbd4d P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 903e5d68fe5e4988a6b54623da741af5 in current term 1: Already voted for candidate fbe14365c42e4c6ba1173d829f997f81 in this term.
I20250831 01:55:25.724270 10215 leader_election.cc:304] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5; no voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:25.724754 10417 raft_consensus.cc:3058] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.727654 10417 raft_consensus.cc:2747] T 4ee86091bc604b6a9d33c56d091fbd4d P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 01:55:25.751531 10410 raft_consensus.cc:491] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.751772 10410 raft_consensus.cc:513] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.753579 10410 leader_election.cc:290] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:25.757542 10451 raft_consensus.cc:491] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.757732 10451 raft_consensus.cc:513] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.758419 10451 leader_election.cc:290] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.759531 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3772a8213fd74d1a995e0c8eed42c08b" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.759765 10008 raft_consensus.cc:2466] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.760144 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3772a8213fd74d1a995e0c8eed42c08b" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.760351 10280 raft_consensus.cc:2466] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:25.760818 10080 leader_election.cc:304] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.762183 10451 raft_consensus.cc:2802] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.762277 10451 raft_consensus.cc:491] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.762391 10451 raft_consensus.cc:3058] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.767803 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd06ceda8134e459d5bb86fac5cdbb8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:25.768086 10146 raft_consensus.cc:2466] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:25.769511 9941 leader_election.cc:304] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.770109 10410 raft_consensus.cc:2802] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.770247 10410 raft_consensus.cc:491] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.770354 10410 raft_consensus.cc:3058] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.770319 10451 raft_consensus.cc:513] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.771396 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3772a8213fd74d1a995e0c8eed42c08b" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:25.771588 10008 raft_consensus.cc:3058] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.773383 10410 raft_consensus.cc:513] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.774365 10008 raft_consensus.cc:2466] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.774415 10410 leader_election.cc:290] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:25.774848 10081 leader_election.cc:304] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.775125 10444 raft_consensus.cc:2802] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.775277 10444 raft_consensus.cc:695] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:25.775583 10444 consensus_queue.cc:237] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.776959 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd06ceda8134e459d5bb86fac5cdbb8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:25.777165 10146 raft_consensus.cc:3058] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.780789 10146 raft_consensus.cc:2466] T 0fd06ceda8134e459d5bb86fac5cdbb8 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:25.781442 9941 leader_election.cc:304] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.785176 10451 leader_election.cc:290] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:25.785847 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3772a8213fd74d1a995e0c8eed42c08b" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.786108 10280 raft_consensus.cc:3058] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.787925 10410 raft_consensus.cc:491] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.788100 10410 raft_consensus.cc:513] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.789578 10280 raft_consensus.cc:2466] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:25.791040 10410 leader_election.cc:290] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:25.791256 10410 raft_consensus.cc:2802] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.791435 10410 raft_consensus.cc:695] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:25.791893 10410 consensus_queue.cc:237] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:25.802803 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1def9b41fbe94b4ba680d0d4afd0d2b8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:25.803133 10146 raft_consensus.cc:2466] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:25.803790 9941 leader_election.cc:304] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:25.804673 10410 raft_consensus.cc:2802] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:25.804791 10410 raft_consensus.cc:491] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:25.804903 10410 raft_consensus.cc:3058] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.806463 10280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd06ceda8134e459d5bb86fac5cdbb8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.806699 10280 raft_consensus.cc:2466] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:25.807046 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd06ceda8134e459d5bb86fac5cdbb8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.807277 10279 raft_consensus.cc:3058] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.810253 10279 raft_consensus.cc:2466] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:25.811386 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1def9b41fbe94b4ba680d0d4afd0d2b8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:25.811656 10279 raft_consensus.cc:2466] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:25.814615 10410 raft_consensus.cc:513] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.815686 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1def9b41fbe94b4ba680d0d4afd0d2b8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:25.815881 10279 raft_consensus.cc:3058] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.818812 10279 raft_consensus.cc:2466] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:25.819339 9942 leader_election.cc:304] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters:
I20250831 01:55:25.819671 10462 raft_consensus.cc:2802] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:25.828104 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1def9b41fbe94b4ba680d0d4afd0d2b8" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:25.828354 10146 raft_consensus.cc:3058] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.828706 10410 leader_election.cc:290] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:25.831321 10146 raft_consensus.cc:2466] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:25.837554 10462 raft_consensus.cc:695] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:25.838150 10462 consensus_queue.cc:237] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:25.870337 9677 catalog_manager.cc:3460] renamed table default.table_0c8cd6f43d2e4afeaf6302f110ebac45 [id=71f91154db86487a915ab787c1495178] in HMS: new name default.table_3bbd369a7fa546ba9996f1935a35e1cd
I20250831 01:55:25.894145 10466 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b3d4b29236443d8a14d363ec2aab7c8)
I20250831 01:55:25.894367 10466 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:25.894483 10417 raft_consensus.cc:491] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:25.894660 10417 raft_consensus.cc:513] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:25.895242 10417 leader_election.cc:290] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:25.895227 10466 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
I20250831 01:55:25.895852 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe4c7b8517747a9a9baef5c25d314f6" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:25.896335 9687 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" is_pre_election: true
I20250831 01:55:25.896431 10008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe4c7b8517747a9a9baef5c25d314f6" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:25.896771 10008 raft_consensus.cc:2391] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 903e5d68fe5e4988a6b54623da741af5 in current term 1: Already voted for candidate fbe14365c42e4c6ba1173d829f997f81 in this term.
I20250831 01:55:25.897496 10213 leader_election.cc:304] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5; no voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:25.898008 10417 raft_consensus.cc:3058] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:25.902351 10417 raft_consensus.cc:2747] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 01:55:25.903188 9803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:25.916563 9803 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:25.916751 9803 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:25.918812 10466 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 01:55:25.929183 9891 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdf49ea1050b4326b1df977f021d3d31 consensus state has no leader
W20250831 01:55:25.930666 9673 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_18f5062db5284a5d9fa9ecc471dc9af7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:25.930888 9670 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b436ecb230034afea5e7f8d551228511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:25.973485 9663 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet fdf49ea1050b4326b1df977f021d3d31 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fdf49ea1050b4326b1df977f021d3d31 consensus state has no leader
W20250831 01:55:25.986912 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:25.987361 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:25.987502 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250831 01:55:25.989073 9676 catalog_manager.cc:3460] renamed table default.table_e8582bab06b043f79df730d33ed076fa [id=b90d279f47a141ed846c461cd398d5e7] in HMS: new name default.table_037da9465b8a41fda06bdeb1cf150d8b
W20250831 01:55:26.000602 10422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:26.000768 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:26.000871 10422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:26.000946 10422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:26.016587 10422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.47.125
--webserver_port=46501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:26.018925 10422 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:26.021378 10422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:26.032272 10478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:26.035358 10479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:26.055660 10482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:26.062543 9674 catalog_manager.cc:3460] renamed table default.table_547c8e6d3c104af9b87f1ae477d22ee0 [id=b9f98af128a1402ca281bd1f8853c194] in HMS: new name default.table_03598ee84555460b9138a48936385e5e
I20250831 01:55:26.063275 10422 server_base.cc:1047] running on GCE node
I20250831 01:55:26.064365 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:26.072073 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:26.073375 10422 hybrid_clock.cc:648] HybridClock initialized: now 1756605326073323 us; error 71 us; skew 500 ppm
I20250831 01:55:26.073796 10422 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:26.081586 10422 webserver.cc:480] Webserver started at http://127.9.47.125:46501/ using document root <none> and password file <none>
I20250831 01:55:26.082635 10422 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:26.082769 10422 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:26.103482 10422 fs_manager.cc:714] Time spent opening directory manager: real 0.019s user 0.015s sys 0.004s
I20250831 01:55:26.107772 10487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:26.109897 10422 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250831 01:55:26.110121 10422 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:26.111202 10422 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:26.138281 9675 catalog_manager.cc:3460] renamed table default.table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5 [id=9584cc0bbc7e437e967c3b8c39ab609b] in HMS: new name default.table_513033900eee43b68b82843e69d0bb74
W20250831 01:55:26.160466 9891 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3772a8213fd74d1a995e0c8eed42c08b consensus state has no leader
W20250831 01:55:26.172466 9891 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 consensus state has no leader
W20250831 01:55:26.173974 9669 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_71f3e7f55e21496389aaf63260dec6a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250831 01:55:26.175202 9668 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e5d7d8712d2d4ec181665ef359124956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 01:55:26.176735 9677 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:38174 (request call id 11) took 1405 ms. Trace:
I20250831 01:55:26.176862 9677 rpcz_store.cc:276] 0831 01:55:24.771032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:24.780358 (+ 9326us) service_pool.cc:225] Handling call
0831 01:55:25.038331 (+257973us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:26.176050 (+1137719us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:26.176396 (+ 346us) catalog_manager.cc:3683] Apply alter schema
0831 01:55:26.176695 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":283093,"Hive Metastore.run_cpu_time_us":1218,"Hive Metastore.run_wall_time_us":548431}
I20250831 01:55:26.186162 10422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:26.187464 10422 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:26.187803 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:26.186038 9677 catalog_manager.cc:5582] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66). New cstate: current_term: 1 leader_uuid: "fbe14365c42e4c6ba1173d829f997f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } }
I20250831 01:55:26.188737 9677 catalog_manager.cc:5674] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085) does not have the latest schema for tablet fdf49ea1050b4326b1df977f021d3d31 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]). Expected version 1 got 0
I20250831 01:55:26.204860 10279 raft_consensus.cc:1273] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Refusing update from remote peer fbe14365c42e4c6ba1173d829f997f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.205544 10008 raft_consensus.cc:1273] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Refusing update from remote peer fbe14365c42e4c6ba1173d829f997f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.206842 10451 consensus_queue.cc:1035] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.207525 10444 consensus_queue.cc:1035] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.209278 9677 catalog_manager.cc:5582] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e9e49e7a864057adb3e685607c9844 (127.9.47.65). New cstate: current_term: 1 leader_uuid: "67e9e49e7a864057adb3e685607c9844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: UNKNOWN } } }
I20250831 01:55:26.209726 9677 catalog_manager.cc:5674] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439) does not have the latest schema for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 (table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]). Expected version 1 got 0
I20250831 01:55:26.210057 9677 catalog_manager.cc:5582] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e9e49e7a864057adb3e685607c9844 (127.9.47.65). New cstate: current_term: 1 leader_uuid: "67e9e49e7a864057adb3e685607c9844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: HEALTHY } } }
I20250831 01:55:26.219496 9677 catalog_manager.cc:5674] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173) does not have the latest schema for tablet fdf49ea1050b4326b1df977f021d3d31 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]). Expected version 1 got 0
I20250831 01:55:26.219817 9677 catalog_manager.cc:5674] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173) does not have the latest schema for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 (table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]). Expected version 1 got 0
I20250831 01:55:26.227864 9677 catalog_manager.cc:5582] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66). New cstate: current_term: 1 leader_uuid: "fbe14365c42e4c6ba1173d829f997f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: HEALTHY } } }
I20250831 01:55:26.228576 9677 catalog_manager.cc:5582] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66). New cstate: current_term: 1 leader_uuid: "fbe14365c42e4c6ba1173d829f997f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: HEALTHY } } }
I20250831 01:55:26.229230 9677 catalog_manager.cc:5582] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66). New cstate: current_term: 1 leader_uuid: "fbe14365c42e4c6ba1173d829f997f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: HEALTHY } } }
I20250831 01:55:26.229696 9677 catalog_manager.cc:5582] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66). New cstate: current_term: 1 leader_uuid: "fbe14365c42e4c6ba1173d829f997f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: HEALTHY } } }
I20250831 01:55:26.230113 9677 catalog_manager.cc:5674] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085) does not have the latest schema for tablet 3772a8213fd74d1a995e0c8eed42c08b (table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]). Expected version 1 got 0
I20250831 01:55:26.230356 9677 catalog_manager.cc:5582] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66). New cstate: current_term: 1 leader_uuid: "fbe14365c42e4c6ba1173d829f997f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: HEALTHY } } }
I20250831 01:55:26.240358 10492 mvcc.cc:204] Tried to move back new op lower bound from 7195055416126390272 to 7195055411948634112. Current Snapshot: MvccSnapshot[applied={T|T < 7195055416126390272}]
I20250831 01:55:26.241197 9677 catalog_manager.cc:5674] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439) does not have the latest schema for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7]). Expected version 1 got 0
I20250831 01:55:26.242494 9677 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_b0242810ea7a4fe3a1d3d8431eca51d2" } modify_external_catalogs: true
I20250831 01:55:26.242733 9677 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_b0242810ea7a4fe3a1d3d8431eca51d2" } modify_external_catalogs: true
I20250831 01:55:26.245277 10501 tablet.cc:1722] T 1fe4c7b8517747a9a9baef5c25d314f6 P fbe14365c42e4c6ba1173d829f997f81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.250319 10280 raft_consensus.cc:1273] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Refusing update from remote peer fbe14365c42e4c6ba1173d829f997f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.251312 10451 consensus_queue.cc:1035] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.254155 10006 raft_consensus.cc:1273] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Refusing update from remote peer fbe14365c42e4c6ba1173d829f997f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.255431 10451 consensus_queue.cc:1035] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.259300 10278 raft_consensus.cc:1273] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Refusing update from remote peer 67e9e49e7a864057adb3e685607c9844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.260078 10462 consensus_queue.cc:1035] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.260320 10146 raft_consensus.cc:1273] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Refusing update from remote peer 67e9e49e7a864057adb3e685607c9844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.261421 10464 consensus_queue.cc:1035] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 01:55:26.265792 9663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20250831 01:55:26.272425 10496 mvcc.cc:204] Tried to move back new op lower bound from 7195055416126390272 to 7195055411948634112. Current Snapshot: MvccSnapshot[applied={T|T < 7195055416126390272}]
I20250831 01:55:26.274485 10276 raft_consensus.cc:1273] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Refusing update from remote peer fbe14365c42e4c6ba1173d829f997f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.275012 10008 raft_consensus.cc:1273] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Refusing update from remote peer fbe14365c42e4c6ba1173d829f997f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 01:55:26.275460 10451 consensus_queue.cc:1035] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.275733 10445 consensus_queue.cc:1035] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.286712 10497 mvcc.cc:204] Tried to move back new op lower bound from 7195055416348512256 to 7195055414637121536. Current Snapshot: MvccSnapshot[applied={T|T < 7195055416348512256}]
I20250831 01:55:26.290326 10501 tablet.cc:1722] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P fbe14365c42e4c6ba1173d829f997f81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.302284 10501 tablet.cc:1722] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.314538 10513 tablet.cc:1722] T 1fe4c7b8517747a9a9baef5c25d314f6 P 67e9e49e7a864057adb3e685607c9844: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.317414 10502 tablet.cc:1722] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 67e9e49e7a864057adb3e685607c9844: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.318773 10511 tablet.cc:1722] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.339291 10508 tablet.cc:1722] T 1fe4c7b8517747a9a9baef5c25d314f6 P 903e5d68fe5e4988a6b54623da741af5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.352344 10502 tablet.cc:1722] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.352417 10512 tablet.cc:1722] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.353124 10516 tablet.cc:1722] T 1def9b41fbe94b4ba680d0d4afd0d2b8 P 903e5d68fe5e4988a6b54623da741af5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.362977 9672 catalog_manager.cc:3460] renamed table default.table_cd781cda81e34e17b5cc76c5689b9600 [id=ad1bd5ef25a343f0ad51d4e99ec44792] in HMS: new name default.table_a3dfa51fdb0a42d390d8b0c78c181087
I20250831 01:55:26.364092 10508 tablet.cc:1722] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.365058 10519 tablet.cc:1722] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:26.413969 10464 raft_consensus.cc:491] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:26.414288 10464 raft_consensus.cc:513] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:26.415467 10464 leader_election.cc:290] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:26.416632 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d596b92ea9d745098c779e349a551e82" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:26.416980 10146 raft_consensus.cc:2466] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:26.417798 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d596b92ea9d745098c779e349a551e82" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:26.418102 10279 raft_consensus.cc:2466] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:26.422181 9941 leader_election.cc:304] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:26.422789 10464 raft_consensus.cc:2802] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:26.422909 10464 raft_consensus.cc:491] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:26.423043 10464 raft_consensus.cc:3058] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:26.426891 10464 raft_consensus.cc:513] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:26.428486 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d596b92ea9d745098c779e349a551e82" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:26.428690 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d596b92ea9d745098c779e349a551e82" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:26.428762 10146 raft_consensus.cc:3058] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:26.428948 10279 raft_consensus.cc:3058] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:26.431581 10146 raft_consensus.cc:2466] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:26.432040 10279 raft_consensus.cc:2466] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:26.432116 9941 leader_election.cc:304] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:26.432507 10462 raft_consensus.cc:2802] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:26.432652 10462 raft_consensus.cc:695] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:26.432948 10462 consensus_queue.cc:237] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:26.434999 10464 leader_election.cc:290] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:26.461866 10422 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.125:46791
I20250831 01:55:26.461867 10560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.125:46791 every 8 connection(s)
I20250831 01:55:26.464145 10422 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
I20250831 01:55:26.472813 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 10422
I20250831 01:55:26.473562 10561 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:26.478638 10561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap starting.
I20250831 01:55:26.490144 10561 log.cc:826] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:26.492746 10451 consensus_queue.cc:1035] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.504678 10561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:26.505399 10561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap complete.
I20250831 01:55:26.513005 10561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:26.514206 10561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3cdeb22a7546b988ab9938f8dc1027, State: Initialized, Role: FOLLOWER
I20250831 01:55:26.516011 10561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:26.524343 10444 consensus_queue.cc:1035] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.558823 10462 consensus_queue.cc:1035] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.569979 10327 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:26.580497 10462 consensus_queue.cc:1035] T 0fd06ceda8134e459d5bb86fac5cdbb8 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.582527 10567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:26.582983 10567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:26.586928 10055 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:26.592206 10561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:26.594604 10193 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:26.594635 10327 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:26.608578 10444 consensus_queue.cc:1035] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.632105 10451 consensus_queue.cc:1035] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.641844 10445 consensus_queue.cc:1035] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.643942 10451 consensus_queue.cc:1035] T 4ee86091bc604b6a9d33c56d091fbd4d P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.738045 10586 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b3d4b29236443d8a14d363ec2aab7c8)
I20250831 01:55:26.738411 10586 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:26.743172 10586 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
I20250831 01:55:26.743579 9687 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" is_pre_election: true
I20250831 01:55:26.755714 10193 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:26.756289 10584 consensus_queue.cc:1035] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.769593 10535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" is_pre_election: true
I20250831 01:55:26.770196 10535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68251650a6ad4c1eba8826b2949c8905 in term 1.
I20250831 01:55:26.770848 9803 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8
I20250831 01:55:26.771380 10586 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 01:55:26.771544 10586 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b3d4b29236443d8a14d363ec2aab7c8)
I20250831 01:55:26.771661 10586 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 1 FOLLOWER]: Advancing to term 2
I20250831 01:55:26.773376 10584 consensus_queue.cc:1035] T d36d4612593e4d8985b160918d54f87a P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:26.776618 10586 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:26.778053 9687 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "0b3d4b29236443d8a14d363ec2aab7c8"
I20250831 01:55:26.778153 10586 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 election: Requested vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
I20250831 01:55:26.779337 10535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
I20250831 01:55:26.779563 10535 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 1 FOLLOWER]: Advancing to term 2
I20250831 01:55:26.791062 10535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68251650a6ad4c1eba8826b2949c8905 in term 2.
I20250831 01:55:26.791685 9803 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8
I20250831 01:55:26.793676 10586 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 01:55:26.820104 10586 raft_consensus.cc:695] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 2 LEADER]: Becoming Leader. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Running, Role: LEADER
I20250831 01:55:26.821031 10586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:26.828199 10535 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Rejecting Update request from peer 0b3d4b29236443d8a14d363ec2aab7c8 for earlier term 1. Current term is 2. Ops: [1.5-1.41]
I20250831 01:55:26.830682 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68251650a6ad4c1eba8826b2949c8905. Latest consensus state: current_term: 2 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:26.831555 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:26.831872 10594 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 01:55:26.834980 10599 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 01:55:26.837250 10598 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 01:55:26.837368 10598 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0b3d4b29236443d8a14d363ec2aab7c8, State: Running, Role: LEADER
I20250831 01:55:26.837810 10598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:26.838688 10598 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250831 01:55:26.839455 10599 catalog_manager.cc:671] Loaded metadata for table default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]
I20250831 01:55:26.840040 10599 catalog_manager.cc:671] Loaded metadata for table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]
I20250831 01:55:26.840574 10599 catalog_manager.cc:671] Loaded metadata for table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]
I20250831 01:55:26.841133 10599 catalog_manager.cc:671] Loaded metadata for table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]
I20250831 01:55:26.841683 10599 catalog_manager.cc:671] Loaded metadata for table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]
I20250831 01:55:26.842247 10599 catalog_manager.cc:671] Loaded metadata for table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]
I20250831 01:55:26.842785 10599 catalog_manager.cc:671] Loaded metadata for table default.table_cd781cda81e34e17b5cc76c5689b9600 [id=ad1bd5ef25a343f0ad51d4e99ec44792]
I20250831 01:55:26.843487 10599 catalog_manager.cc:671] Loaded metadata for table default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]
I20250831 01:55:26.844066 10599 catalog_manager.cc:671] Loaded metadata for table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7]
I20250831 01:55:26.844585 10599 catalog_manager.cc:671] Loaded metadata for table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]
I20250831 01:55:26.847429 10599 tablet_loader.cc:96] loaded metadata for tablet 0fd06ceda8134e459d5bb86fac5cdbb8 (table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb])
I20250831 01:55:26.847981 10599 tablet_loader.cc:96] loaded metadata for tablet 1ba0bc1eadc64d219b49437241867a85 (table default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8])
I20250831 01:55:26.848511 10599 tablet_loader.cc:96] loaded metadata for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 (table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b])
I20250831 01:55:26.849027 10599 tablet_loader.cc:96] loaded metadata for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7])
I20250831 01:55:26.849471 10599 tablet_loader.cc:96] loaded metadata for tablet 3772a8213fd74d1a995e0c8eed42c08b (table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194])
I20250831 01:55:26.849972 10599 tablet_loader.cc:96] loaded metadata for tablet 4ee86091bc604b6a9d33c56d091fbd4d (table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5])
I20250831 01:55:26.850380 10599 tablet_loader.cc:96] loaded metadata for tablet 5cbcda68336141109d5e3c96f97454a5 (table default.table_cd781cda81e34e17b5cc76c5689b9600 [id=ad1bd5ef25a343f0ad51d4e99ec44792])
I20250831 01:55:26.851217 10599 tablet_loader.cc:96] loaded metadata for tablet d36d4612593e4d8985b160918d54f87a (table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa])
I20250831 01:55:26.851920 10599 tablet_loader.cc:96] loaded metadata for tablet d596b92ea9d745098c779e349a551e82 (table default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a])
I20250831 01:55:26.852360 10599 tablet_loader.cc:96] loaded metadata for tablet fdf49ea1050b4326b1df977f021d3d31 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178])
I20250831 01:55:26.852730 10599 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 01:55:26.854359 10599 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:26.854488 10599 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 01:55:26.856412 10599 catalog_manager.cc:1506] Loading token signing keys...
I20250831 01:55:26.857897 10599 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Loaded TSK: 0
I20250831 01:55:26.859192 10599 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 01:55:26.860971 10599 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250831 01:55:26.862612 10599 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 14
I20250831 01:55:26.882340 10604 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:26.882484 10604 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: loading cluster ID for follower catalog manager: success
I20250831 01:55:26.887733 10604 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: acquiring CA information for follower catalog manager: success
I20250831 01:55:26.891587 10604 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:27.082110 9671 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]
I20250831 01:55:27.133174 9673 catalog_manager.cc:2149] created HMS catalog entry for table [id=ade4255a8f3e4cb0a9878ddc5ad33d44]
W20250831 01:55:27.134282 9673 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER
I20250831 01:55:27.150012 10535 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Refusing update from remote peer 68251650a6ad4c1eba8826b2949c8905: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20250831 01:55:27.153891 10586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 01:55:27.159600 9687 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 2 FOLLOWER]: Refusing update from remote peer 68251650a6ad4c1eba8826b2949c8905: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20250831 01:55:27.160436 10596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250831 01:55:27.175103 10598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68251650a6ad4c1eba8826b2949c8905. Latest consensus state: current_term: 2 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:27.175372 10598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:27.178442 9893 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:27.182475 10598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:27.182724 10598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:27.195266 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:27.195547 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:27.197853 9873 catalog_manager.cc:2747] Deleting table default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8] in response to Hive Metastore notification log event 16
I20250831 01:55:27.202966 9670 catalog_manager.cc:2149] created HMS catalog entry for table [id=99d0e7e730954cc6abb603bb85b1db18]
W20250831 01:55:27.203802 9670 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER
I20250831 01:55:27.207856 9873 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Sending DeleteTablet for 3 replicas of tablet 1ba0bc1eadc64d219b49437241867a85
I20250831 01:55:27.287695 9669 catalog_manager.cc:2149] created HMS catalog entry for table [id=5dc2dd483179499eafb15a91231b58ce]
W20250831 01:55:27.288606 9669 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER
I20250831 01:55:27.292337 10258 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ba0bc1eadc64d219b49437241867a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:48122
I20250831 01:55:27.297693 10126 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ba0bc1eadc64d219b49437241867a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:42120
I20250831 01:55:27.303745 10619 tablet_replica.cc:331] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:27.304605 10619 raft_consensus.cc:2241] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.305244 10619 raft_consensus.cc:2270] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.306352 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ba0bc1eadc64d219b49437241867a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:38384
I20250831 01:55:27.318274 10621 tablet_replica.cc:331] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:27.318537 10621 raft_consensus.cc:2241] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Raft consensus shutting down.
I20250831 01:55:27.319185 10621 raft_consensus.cc:2270] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.317744 10619 ts_tablet_manager.cc:1905] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.322173 10621 ts_tablet_manager.cc:1905] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.322417 10623 tablet_replica.cc:331] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:27.322695 10623 raft_consensus.cc:2241] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.322935 10623 raft_consensus.cc:2270] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.324745 10623 ts_tablet_manager.cc:1905] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.342805 10621 ts_tablet_manager.cc:1918] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:27.342968 10621 log.cc:1199] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/1ba0bc1eadc64d219b49437241867a85
I20250831 01:55:27.343977 10191 heartbeater.cc:499] Master 127.9.47.124:37481 was elected leader, sending a full tablet report...
I20250831 01:55:27.349165 10621 ts_tablet_manager.cc:1939] T 1ba0bc1eadc64d219b49437241867a85 P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:27.349308 9817 catalog_manager.cc:5674] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085) does not have the latest schema for tablet 5cbcda68336141109d5e3c96f97454a5 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]). Expected version 1 got 0
I20250831 01:55:27.352526 10523 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:53185; Asking this server to re-register.
I20250831 01:55:27.353402 10524 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:42319; Asking this server to re-register.
I20250831 01:55:27.354372 10525 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:41665; Asking this server to re-register.
I20250831 01:55:27.355224 10327 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:27.355830 10193 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:27.356699 10055 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:27.358613 10609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68251650a6ad4c1eba8826b2949c8905. Latest consensus state: current_term: 2 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:27.358989 10609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:27.359174 9817 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178] alter complete (version 1)
I20250831 01:55:27.361378 10525 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:27.363166 10524 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:27.366022 9817 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7] alter complete (version 1)
I20250831 01:55:27.366351 10524 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:27.367215 9801 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet 1ba0bc1eadc64d219b49437241867a85 (table default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]) successfully deleted
I20250831 01:55:27.372959 9817 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194] alter complete (version 1)
I20250831 01:55:27.379709 10632 tablet.cc:1722] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:27.380059 9817 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b] alter complete (version 1)
I20250831 01:55:27.381239 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ba0bc1eadc64d219b49437241867a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:42120
I20250831 01:55:27.381826 10633 tablet.cc:1722] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250831 01:55:27.382606 9801 catalog_manager.cc:4895] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): delete failed for tablet 1ba0bc1eadc64d219b49437241867a85 because the tablet was not found. No further retry: Not found: Tablet not found: 1ba0bc1eadc64d219b49437241867a85
I20250831 01:55:27.384482 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:27.384727 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:27.392549 10609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:27.392786 10609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:27.403280 10623 ts_tablet_manager.cc:1918] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:27.403542 10623 log.cc:1199] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/1ba0bc1eadc64d219b49437241867a85
I20250831 01:55:27.404165 10623 ts_tablet_manager.cc:1939] T 1ba0bc1eadc64d219b49437241867a85 P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:27.405409 9803 catalog_manager.cc:4928] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): tablet 1ba0bc1eadc64d219b49437241867a85 (table default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]) successfully deleted
I20250831 01:55:27.412070 10611 mvcc.cc:204] Tried to move back new op lower bound from 7195055419996643328 to 7195055418665033728. Current Snapshot: MvccSnapshot[applied={T|T < 7195055416246800384}]
I20250831 01:55:27.414595 10634 tablet.cc:1722] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250831 01:55:27.416298 10325 heartbeater.cc:499] Master 127.9.47.124:37481 was elected leader, sending a full tablet report...
I20250831 01:55:27.420670 10619 ts_tablet_manager.cc:1918] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:27.420843 10619 log.cc:1199] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/1ba0bc1eadc64d219b49437241867a85
I20250831 01:55:27.421356 10619 ts_tablet_manager.cc:1939] T 1ba0bc1eadc64d219b49437241867a85 P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:27.422631 9802 catalog_manager.cc:4928] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): tablet 1ba0bc1eadc64d219b49437241867a85 (table default.table_e32aefba1e29406f91d7559a02f1323f [id=0c52c96929564376a2419751be04d5b8]) successfully deleted
I20250831 01:55:27.428052 9817 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792] alter complete (version 1)
I20250831 01:55:27.430393 10258 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ba0bc1eadc64d219b49437241867a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:48122
W20250831 01:55:27.431217 9802 catalog_manager.cc:4895] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): delete failed for tablet 1ba0bc1eadc64d219b49437241867a85 because the tablet was not found. No further retry: Not found: Tablet not found: 1ba0bc1eadc64d219b49437241867a85
I20250831 01:55:27.433501 10053 heartbeater.cc:499] Master 127.9.47.124:37481 was elected leader, sending a full tablet report...
I20250831 01:55:27.440729 9815 catalog_manager.cc:5582] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e9e49e7a864057adb3e685607c9844 (127.9.47.65). New cstate: current_term: 1 leader_uuid: "67e9e49e7a864057adb3e685607c9844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } health_report { overall_health: UNKNOWN } } }
I20250831 01:55:27.447574 9668 catalog_manager.cc:2149] created HMS catalog entry for table [id=2f6da2a74ea0463397b0da181cd0b2c1]
W20250831 01:55:27.448496 9668 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER
W20250831 01:55:27.460384 9891 catalog_manager.cc:3897] Illegal state: an error occurred while updating the sys-catalog: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER
W20250831 01:55:27.460831 9891 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 15 ALTER_TABLE default.table_a3dfa51fdb0a42d390d8b0c78c181087: Illegal state: an error occurred while updating the sys-catalog: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER
W20250831 01:55:27.461095 9891 hms_notification_log_listener.cc:143] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER
I20250831 01:55:27.461436 9676 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:38174 (request call id 14) took 2581 ms. Trace:
I20250831 01:55:27.461555 9676 rpcz_store.cc:276] 0831 01:55:24.879908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:24.885939 (+ 6031us) service_pool.cc:225] Handling call
0831 01:55:25.159172 (+273233us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:27.461391 (+2302219us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":715997,"Hive Metastore.run_cpu_time_us":714,"Hive Metastore.run_wall_time_us":113508}
W20250831 01:55:27.462184 9676 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.47.66:36923 (request call id 14) took 1217 ms (client timeout 1000 ms). Trace:
W20250831 01:55:27.462301 9676 rpcz_store.cc:269] 0831 01:55:26.245164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.461933 (+1216769us) service_pool.cc:212] Skipping call since client already timed out
0831 01:55:27.462139 (+ 206us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 01:55:27.462585 9676 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.47.67:36619 (request call id 8) took 1197 ms (client timeout 1000 ms). Trace:
W20250831 01:55:27.462688 9676 rpcz_store.cc:269] 0831 01:55:26.264784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.462521 (+1197737us) service_pool.cc:212] Skipping call since client already timed out
0831 01:55:27.462553 (+ 32us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 01:55:27.462900 9676 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.47.65:52921 (request call id 13) took 1139 ms (client timeout 1000 ms). Trace:
W20250831 01:55:27.462994 9676 rpcz_store.cc:269] 0831 01:55:26.323947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.462842 (+1138895us) service_pool.cc:212] Skipping call since client already timed out
0831 01:55:27.462873 (+ 31us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 01:55:27.467218 10334 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.47.126:37509)
I20250831 01:55:27.479857 9674 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:38174 (request call id 12) took 2689 ms. Trace:
I20250831 01:55:27.480032 9674 rpcz_store.cc:276] 0831 01:55:24.790808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:24.790877 (+ 69us) service_pool.cc:225] Handling call
0831 01:55:24.800219 (+ 9342us) spinlock_profiling.cc:243] Waited 8.99 ms on lock 0x616000040f54. stack: 00007fef88a4d7d6 00007fef88a4d590 00007fef87a403ad 0000000000972d94 0000000000972d5d 00007fef9781814d 00007fef97818243 00007fef944d7941 00007fef9c062a5d 00007fef9c1ee126 00007fef93027126 00007fef93026e55 00007fef8f7fb2ce 00007fef8f7fa1ae 00007fef8f8003e5 00007fef8f800d0f
0831 01:55:25.159327 (+359108us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:27.479796 (+2320469us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":829485,"Hive Metastore.run_cpu_time_us":690,"Hive Metastore.run_wall_time_us":73397,"spinlock_wait_cycles":20674304}
I20250831 01:55:27.482492 9674 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 27) took 2338 ms. Trace:
I20250831 01:55:27.482642 9674 rpcz_store.cc:276] 0831 01:55:25.144505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.482089 (+2337584us) service_pool.cc:225] Handling call
0831 01:55:27.482450 (+ 361us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.483258 9674 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:38174 (request call id 28) took 2333 ms. Trace:
I20250831 01:55:27.483385 9674 rpcz_store.cc:276] 0831 01:55:25.149738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.482909 (+2333171us) service_pool.cc:225] Handling call
0831 01:55:27.483223 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.483644 9674 master_service.cc:948] ReplaceTablet: received request to replace tablet 5cbcda68336141109d5e3c96f97454a5 from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:27.483994 9674 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:38174 (request call id 29) took 2270 ms. Trace:
I20250831 01:55:27.484087 9674 rpcz_store.cc:276] 0831 01:55:25.213827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.483589 (+2269762us) service_pool.cc:225] Handling call
0831 01:55:27.483940 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.484371 9674 master_service.cc:948] ReplaceTablet: received request to replace tablet 5cbcda68336141109d5e3c96f97454a5 from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:27.484686 9674 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:38174 (request call id 30) took 2264 ms. Trace:
I20250831 01:55:27.484787 9674 rpcz_store.cc:276] 0831 01:55:25.219859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.484288 (+2264429us) service_pool.cc:225] Handling call
0831 01:55:27.484654 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.485237 9674 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 31) took 2254 ms. Trace:
I20250831 01:55:27.485339 9674 rpcz_store.cc:276] 0831 01:55:25.231297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.484929 (+2253632us) service_pool.cc:225] Handling call
0831 01:55:27.485207 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.485560 9674 master_service.cc:948] ReplaceTablet: received request to replace tablet 3772a8213fd74d1a995e0c8eed42c08b from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:27.487332 9675 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:38174 (request call id 13) took 2636 ms. Trace:
I20250831 01:55:27.487474 9675 rpcz_store.cc:276] 0831 01:55:24.850402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:24.850461 (+ 59us) service_pool.cc:225] Handling call
0831 01:55:25.159589 (+309128us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:27.487293 (+2327704us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":902800,"Hive Metastore.run_cpu_time_us":737,"Hive Metastore.run_wall_time_us":75673}
I20250831 01:55:27.487807 9675 master_service.cc:948] ReplaceTablet: received request to replace tablet 1ba0bc1eadc64d219b49437241867a85 from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:27.488149 9675 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:38174 (request call id 33) took 2235 ms. Trace:
I20250831 01:55:27.488234 9675 rpcz_store.cc:276] 0831 01:55:25.252399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.487752 (+2235353us) service_pool.cc:225] Handling call
0831 01:55:27.488098 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.488474 9675 master_service.cc:948] ReplaceTablet: received request to replace tablet fdf49ea1050b4326b1df977f021d3d31 from {username='slave'} at 127.0.0.1:38174
I20250831 01:55:27.488777 9675 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:38174 (request call id 34) took 2235 ms. Trace:
I20250831 01:55:27.488888 9675 rpcz_store.cc:276] 0831 01:55:25.253180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.488398 (+2235218us) service_pool.cc:225] Handling call
0831 01:55:27.488746 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.489296 9675 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:38174 (request call id 35) took 2219 ms. Trace:
I20250831 01:55:27.489405 9675 rpcz_store.cc:276] 0831 01:55:25.269909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.489031 (+2219122us) service_pool.cc:225] Handling call
0831 01:55:27.489265 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.491003 9672 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:38174 (request call id 20) took 2537 ms. Trace:
I20250831 01:55:27.491110 9672 rpcz_store.cc:276] 0831 01:55:24.953449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:25.041617 (+ 88168us) service_pool.cc:225] Handling call
0831 01:55:25.930030 (+888413us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:27.490371 (+1560341us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:27.490627 (+ 256us) catalog_manager.cc:3683] Apply alter schema
0831 01:55:27.490943 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":211980,"Hive Metastore.run_cpu_time_us":619,"Hive Metastore.run_wall_time_us":220583}
I20250831 01:55:27.491700 9672 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:38174 (request call id 37) took 2155 ms. Trace:
I20250831 01:55:27.491798 9672 rpcz_store.cc:276] 0831 01:55:25.335948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.491393 (+2155445us) service_pool.cc:225] Handling call
0831 01:55:27.491648 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.492297 9672 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 38) took 2142 ms. Trace:
I20250831 01:55:27.492399 9672 rpcz_store.cc:276] 0831 01:55:25.350235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.491955 (+2141720us) service_pool.cc:225] Handling call
0831 01:55:27.492246 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.492887 9672 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 39) took 2026 ms. Trace:
I20250831 01:55:27.492978 9672 rpcz_store.cc:276] 0831 01:55:25.466655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.492550 (+2025895us) service_pool.cc:225] Handling call
0831 01:55:27.492832 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.493523 9672 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 40) took 1957 ms. Trace:
I20250831 01:55:27.493661 9672 rpcz_store.cc:276] 0831 01:55:25.536404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.493123 (+1956719us) service_pool.cc:225] Handling call
0831 01:55:27.493469 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.494235 9672 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:38174 (request call id 41) took 1111 ms. Trace:
I20250831 01:55:27.494325 9672 rpcz_store.cc:276] 0831 01:55:26.382981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.493912 (+1110931us) service_pool.cc:225] Handling call
0831 01:55:27.494182 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 01:55:27.499455 10334 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.47.126:37509)
W20250831 01:55:27.499963 10334 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.47.126:37509)
W20250831 01:55:27.500320 10334 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.47.126:37509)
I20250831 01:55:27.504387 9675 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 36) took 2209 ms. Trace:
I20250831 01:55:27.504535 9675 rpcz_store.cc:276] 0831 01:55:25.294637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.489553 (+2194916us) service_pool.cc:225] Handling call
0831 01:55:27.504329 (+ 14776us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.516481 9671 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:38174 (request call id 23) took 2484 ms. Trace:
I20250831 01:55:27.516618 9671 rpcz_store.cc:276] 0831 01:55:25.032313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:25.148600 (+116287us) service_pool.cc:225] Handling call
0831 01:55:25.930640 (+782040us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:27.516410 (+1585770us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":432088,"Hive Metastore.run_cpu_time_us":426,"Hive Metastore.run_wall_time_us":719074}
I20250831 01:55:27.528604 9674 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:38174 (request call id 32) took 2276 ms. Trace:
I20250831 01:55:27.528842 9674 rpcz_store.cc:276] 0831 01:55:25.251996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:27.485510 (+2233514us) service_pool.cc:225] Handling call
0831 01:55:27.527935 (+ 42425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 01:55:27.545282 9812 master_service.cc:948] ReplaceTablet: received request to replace tablet 5cbcda68336141109d5e3c96f97454a5 from {username='slave'} at 127.0.0.1:34126
I20250831 01:55:27.545727 9813 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_519e74debc8b417c8fe427b8b4c3bd29" } modify_external_catalogs: true
I20250831 01:55:27.546061 9813 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_519e74debc8b417c8fe427b8b4c3bd29" } modify_external_catalogs: true
I20250831 01:55:27.545378 9811 master_service.cc:948] ReplaceTablet: received request to replace tablet 5cbcda68336141109d5e3c96f97454a5 from {username='slave'} at 127.0.0.1:34126
I20250831 01:55:27.546371 9808 master_service.cc:948] ReplaceTablet: received request to replace tablet 1ba0bc1eadc64d219b49437241867a85 from {username='slave'} at 127.0.0.1:34126
I20250831 01:55:27.555516 9817 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_e8582bab06b043f79df730d33ed076fa" } new_table_name: "default.Table_037da9465b8a41fda06bdeb1cf150d8b" modify_external_catalogs: true
I20250831 01:55:27.555759 9673 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=ade4255a8f3e4cb0a9878ddc5ad33d44]
I20250831 01:55:27.556277 9673 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 21) took 2585 ms. Trace:
I20250831 01:55:27.556697 9873 catalog_manager.cc:2747] Deleting table default.table_18f5062db5284a5d9fa9ecc471dc9af7 [id=ade4255a8f3e4cb0a9878ddc5ad33d44] in response to Hive Metastore notification log event 21
W20250831 01:55:27.556948 9873 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 21 DROP_TABLE default.table_18f5062db5284a5d9fa9ecc471dc9af7: Not found: the table does not exist: table_id: "ade4255a8f3e4cb0a9878ddc5ad33d44" table_name: "default.table_18f5062db5284a5d9fa9ecc471dc9af7"
I20250831 01:55:27.556371 9673 rpcz_store.cc:276] 0831 01:55:24.970827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:25.050126 (+ 79299us) service_pool.cc:225] Handling call
0831 01:55:25.930797 (+880671us) catalog_manager.cc:2076] Acquired catalog manager lock
0831 01:55:25.931362 (+ 565us) catalog_manager.cc:2122] Created new table and tablet info
0831 01:55:27.133130 (+1201768us) catalog_manager.cc:2148] Created new table in HMS catalog
0831 01:55:27.134111 (+ 981us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.134115 (+ 4us) write_op.cc:454] Released schema lock
0831 01:55:27.134406 (+ 291us) catalog_manager.cc:2156] Rolling back HMS table creation
0831 01:55:27.556203 (+421797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 01:55:27.133996 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER)
0831 01:55:27.134013 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.134020 (+ 7us) write_op.cc:454] Released schema lock
0831 01:55:27.134025 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1473400,"Hive Metastore.run_cpu_time_us":594,"Hive Metastore.run_wall_time_us":149252,"spinlock_wait_cycles":8320,"child_traces":[["op",{}]]}
I20250831 01:55:27.561067 9808 master_service.cc:948] ReplaceTablet: received request to replace tablet fdf49ea1050b4326b1df977f021d3d31 from {username='slave'} at 127.0.0.1:34126
I20250831 01:55:27.554718 9809 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_f1e0d6faa3d74ec2b9bf4b6e0b171aa5" } new_table_name: "default.Table_513033900eee43b68b82843e69d0bb74" modify_external_catalogs: true
I20250831 01:55:27.554023 9810 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34126:
name: "default.Table_a91e9d30219f48539efe3ebbbf819119"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:27.567768 9812 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Sending DeleteTablet for 3 replicas of tablet 5cbcda68336141109d5e3c96f97454a5
I20250831 01:55:27.568302 9812 catalog_manager.cc:6652] ReplaceTablet: tablet 5cbcda68336141109d5e3c96f97454a5 deleted and replaced by tablet c2b779e787cc4e808f7b74558e733471
I20250831 01:55:27.568643 9812 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_54383364d8c14085a4963804ac2d6ba0" } modify_external_catalogs: true
I20250831 01:55:27.568890 9812 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_54383364d8c14085a4963804ac2d6ba0" } modify_external_catalogs: true
I20250831 01:55:27.569610 9811 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_2faa27c5430c42e7b69a131d36238fdd" } modify_external_catalogs: true
I20250831 01:55:27.569823 9811 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_2faa27c5430c42e7b69a131d36238fdd" } modify_external_catalogs: true
I20250831 01:55:27.570009 9816 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_547c8e6d3c104af9b87f1ae477d22ee0" } new_table_name: "default.Table_03598ee84555460b9138a48936385e5e" modify_external_catalogs: true
I20250831 01:55:27.570485 9814 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34126:
name: "default.Table_94d522ff92ac4ee7ae3906c381bfc49a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:27.573096 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cbcda68336141109d5e3c96f97454a5 with delete_type TABLET_DATA_DELETED (replaced by tablet c2b779e787cc4e808f7b74558e733471) from {username='slave'} at 127.0.0.1:38384
I20250831 01:55:27.573374 10623 tablet_replica.cc:331] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:27.573619 10623 raft_consensus.cc:2241] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.573859 10623 raft_consensus.cc:2270] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.574190 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cbcda68336141109d5e3c96f97454a5 with delete_type TABLET_DATA_DELETED (replaced by tablet c2b779e787cc4e808f7b74558e733471) from {username='slave'} at 127.0.0.1:42120
I20250831 01:55:27.574750 10621 tablet_replica.cc:331] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:27.574956 10621 raft_consensus.cc:2241] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Raft consensus shutting down.
I20250831 01:55:27.575302 10621 raft_consensus.cc:2270] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.576360 10623 ts_tablet_manager.cc:1905] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.578629 10621 ts_tablet_manager.cc:1905] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.582602 10623 ts_tablet_manager.cc:1918] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:27.582827 10623 log.cc:1199] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/5cbcda68336141109d5e3c96f97454a5
I20250831 01:55:27.583330 10623 ts_tablet_manager.cc:1939] T 5cbcda68336141109d5e3c96f97454a5 P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:27.585142 9803 catalog_manager.cc:4928] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): tablet 5cbcda68336141109d5e3c96f97454a5 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]) successfully deleted
I20250831 01:55:27.585757 9808 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Sending DeleteTablet for 3 replicas of tablet fdf49ea1050b4326b1df977f021d3d31
I20250831 01:55:27.586105 10621 ts_tablet_manager.cc:1918] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:27.586278 9808 catalog_manager.cc:6652] ReplaceTablet: tablet fdf49ea1050b4326b1df977f021d3d31 deleted and replaced by tablet 0245596193a346d3aeb2d0aee88a6103
I20250831 01:55:27.586278 10621 log.cc:1199] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/5cbcda68336141109d5e3c96f97454a5
I20250831 01:55:27.586783 10621 ts_tablet_manager.cc:1939] T 5cbcda68336141109d5e3c96f97454a5 P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:27.586673 9808 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34126:
name: "default.Table_a72e022123de434a95f24db4c575dd6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:27.587821 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet fdf49ea1050b4326b1df977f021d3d31 with delete_type TABLET_DATA_DELETED (replaced by tablet 0245596193a346d3aeb2d0aee88a6103) from {username='slave'} at 127.0.0.1:38384
I20250831 01:55:27.587960 9801 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet 5cbcda68336141109d5e3c96f97454a5 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]) successfully deleted
I20250831 01:55:27.588145 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet fdf49ea1050b4326b1df977f021d3d31 with delete_type TABLET_DATA_DELETED (replaced by tablet 0245596193a346d3aeb2d0aee88a6103) from {username='slave'} at 127.0.0.1:42120
I20250831 01:55:27.588835 10258 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cbcda68336141109d5e3c96f97454a5 with delete_type TABLET_DATA_DELETED (replaced by tablet c2b779e787cc4e808f7b74558e733471) from {username='slave'} at 127.0.0.1:48122
I20250831 01:55:27.589144 10619 tablet_replica.cc:331] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:27.589394 10619 raft_consensus.cc:2241] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.589692 10619 raft_consensus.cc:2270] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.590206 10258 tablet_service.cc:1515] Processing DeleteTablet for tablet fdf49ea1050b4326b1df977f021d3d31 with delete_type TABLET_DATA_DELETED (replaced by tablet 0245596193a346d3aeb2d0aee88a6103) from {username='slave'} at 127.0.0.1:48122
I20250831 01:55:27.590586 10619 ts_tablet_manager.cc:1905] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.593248 10623 tablet_replica.cc:331] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:27.593442 10623 raft_consensus.cc:2241] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.593664 10623 raft_consensus.cc:2270] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.594270 10623 ts_tablet_manager.cc:1905] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.597564 10619 ts_tablet_manager.cc:1918] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:27.597729 10619 log.cc:1199] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/5cbcda68336141109d5e3c96f97454a5
I20250831 01:55:27.598183 10619 ts_tablet_manager.cc:1939] T 5cbcda68336141109d5e3c96f97454a5 P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:27.605580 9815 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34126:
name: "default.Table_86b82dc71b1c42cdb1f2c620c2e2771a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:27.605697 10462 consensus_queue.cc:1035] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:27.606964 10621 tablet_replica.cc:331] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:27.607177 10621 raft_consensus.cc:2241] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Raft consensus shutting down.
I20250831 01:55:27.607512 10621 raft_consensus.cc:2270] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.607841 9802 catalog_manager.cc:4928] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): tablet 5cbcda68336141109d5e3c96f97454a5 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]) successfully deleted
I20250831 01:55:27.608141 10621 ts_tablet_manager.cc:1905] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.613178 10619 tablet_replica.cc:331] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:27.613466 10619 raft_consensus.cc:2241] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.613831 10619 raft_consensus.cc:2270] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.614528 10619 ts_tablet_manager.cc:1905] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.630544 10125 tablet_service.cc:1468] Processing CreateTablet for tablet c2b779e787cc4e808f7b74558e733471 (DEFAULT_TABLE table=default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.631027 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2b779e787cc4e808f7b74558e733471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:27.634940 10258 tablet_service.cc:1468] Processing CreateTablet for tablet c2b779e787cc4e808f7b74558e733471 (DEFAULT_TABLE table=default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.635425 10258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2b779e787cc4e808f7b74558e733471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:27.638621 9984 tablet_service.cc:1468] Processing CreateTablet for tablet c2b779e787cc4e808f7b74558e733471 (DEFAULT_TABLE table=default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.639092 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2b779e787cc4e808f7b74558e733471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:27.640216 10619 ts_tablet_manager.cc:1918] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:27.640362 10619 log.cc:1199] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/fdf49ea1050b4326b1df977f021d3d31
I20250831 01:55:27.640811 10619 ts_tablet_manager.cc:1939] T fdf49ea1050b4326b1df977f021d3d31 P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:27.646121 9802 catalog_manager.cc:4928] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): tablet fdf49ea1050b4326b1df977f021d3d31 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]) successfully deleted
I20250831 01:55:27.659724 10646 tablet_bootstrap.cc:492] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:27.662148 10646 tablet_bootstrap.cc:654] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:27.664402 10462 consensus_queue.cc:1035] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:27.663659 10645 tablet_bootstrap.cc:492] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:27.667769 10645 tablet_bootstrap.cc:654] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:27.668665 10643 tablet_bootstrap.cc:492] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:27.669025 9984 tablet_service.cc:1468] Processing CreateTablet for tablet 0245596193a346d3aeb2d0aee88a6103 (DEFAULT_TABLE table=default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.669566 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0245596193a346d3aeb2d0aee88a6103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:27.679656 10643 tablet_bootstrap.cc:654] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:27.682896 10623 ts_tablet_manager.cc:1918] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:27.686914 10623 log.cc:1199] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/fdf49ea1050b4326b1df977f021d3d31
I20250831 01:55:27.687669 10623 ts_tablet_manager.cc:1939] T fdf49ea1050b4326b1df977f021d3d31 P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:27.689252 9803 catalog_manager.cc:4928] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): tablet fdf49ea1050b4326b1df977f021d3d31 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]) successfully deleted
I20250831 01:55:27.699179 10643 tablet_bootstrap.cc:492] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:27.699555 10643 ts_tablet_manager.cc:1397] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20250831 01:55:27.700908 10643 raft_consensus.cc:357] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:27.701413 10643 raft_consensus.cc:383] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:27.701608 10643 raft_consensus.cc:738] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:27.702104 10643 consensus_queue.cc:260] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:27.702991 10643 ts_tablet_manager.cc:1428] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250831 01:55:27.703730 10125 tablet_service.cc:1468] Processing CreateTablet for tablet 0245596193a346d3aeb2d0aee88a6103 (DEFAULT_TABLE table=default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.704221 10125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0245596193a346d3aeb2d0aee88a6103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:27.707810 10258 tablet_service.cc:1468] Processing CreateTablet for tablet 0245596193a346d3aeb2d0aee88a6103 (DEFAULT_TABLE table=default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.708644 10258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0245596193a346d3aeb2d0aee88a6103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:27.710124 9670 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=99d0e7e730954cc6abb603bb85b1db18]
I20250831 01:55:27.710508 9670 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 22) took 2719 ms. Trace:
I20250831 01:55:27.710615 9670 rpcz_store.cc:276] 0831 01:55:24.991174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:25.093006 (+101832us) service_pool.cc:225] Handling call
0831 01:55:25.931004 (+837998us) catalog_manager.cc:2076] Acquired catalog manager lock
0831 01:55:25.931436 (+ 432us) catalog_manager.cc:2122] Created new table and tablet info
0831 01:55:27.202922 (+1271486us) catalog_manager.cc:2148] Created new table in HMS catalog
0831 01:55:27.203672 (+ 750us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.203676 (+ 4us) write_op.cc:454] Released schema lock
0831 01:55:27.203898 (+ 222us) catalog_manager.cc:2156] Rolling back HMS table creation
0831 01:55:27.710438 (+506540us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 01:55:27.203603 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER)
0831 01:55:27.203615 (+ 12us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.203621 (+ 6us) write_op.cc:454] Released schema lock
0831 01:55:27.203626 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1553283,"Hive Metastore.run_cpu_time_us":466,"Hive Metastore.run_wall_time_us":223723,"mutex_wait_us":24,"spinlock_wait_cycles":9728,"child_traces":[["op",{}]]}
I20250831 01:55:27.723449 10621 ts_tablet_manager.cc:1918] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:27.723621 10621 log.cc:1199] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/fdf49ea1050b4326b1df977f021d3d31
I20250831 01:55:27.724143 10621 ts_tablet_manager.cc:1939] T fdf49ea1050b4326b1df977f021d3d31 P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:27.725327 9801 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet fdf49ea1050b4326b1df977f021d3d31 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]) successfully deleted
I20250831 01:55:27.728976 10643 tablet_bootstrap.cc:492] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:27.744421 10643 tablet_bootstrap.cc:654] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:27.747598 10645 tablet_bootstrap.cc:492] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:27.747800 10645 ts_tablet_manager.cc:1397] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.084s user 0.000s sys 0.004s
I20250831 01:55:27.748539 10643 tablet_bootstrap.cc:492] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:27.748652 10645 raft_consensus.cc:357] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:27.748878 10645 raft_consensus.cc:383] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:27.748986 10645 raft_consensus.cc:738] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:27.749004 10643 ts_tablet_manager.cc:1397] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20250831 01:55:27.749284 10645 consensus_queue.cc:260] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:27.749979 10645 ts_tablet_manager.cc:1428] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250831 01:55:27.750280 10645 tablet_bootstrap.cc:492] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:27.750705 10643 raft_consensus.cc:357] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.751052 10643 raft_consensus.cc:383] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:27.751242 10643 raft_consensus.cc:738] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:27.751686 10643 consensus_queue.cc:260] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.751978 10645 tablet_bootstrap.cc:654] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:27.754873 10643 ts_tablet_manager.cc:1428] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250831 01:55:27.762075 10646 tablet_bootstrap.cc:492] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:27.762270 10646 ts_tablet_manager.cc:1397] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.007s
I20250831 01:55:27.763152 10646 raft_consensus.cc:357] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:27.763366 10646 raft_consensus.cc:383] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:27.763484 10646 raft_consensus.cc:738] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:27.763778 10646 consensus_queue.cc:260] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:27.766826 10646 ts_tablet_manager.cc:1428] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250831 01:55:27.769258 10646 tablet_bootstrap.cc:492] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:27.771052 10646 tablet_bootstrap.cc:654] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:27.786123 10646 tablet_bootstrap.cc:492] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:27.786335 10646 ts_tablet_manager.cc:1397] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20250831 01:55:27.787248 10646 raft_consensus.cc:357] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.787479 10646 raft_consensus.cc:383] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:27.787595 10646 raft_consensus.cc:738] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:27.788614 10645 tablet_bootstrap.cc:492] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:27.788791 10645 ts_tablet_manager.cc:1397] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250831 01:55:27.789626 10645 raft_consensus.cc:357] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.789855 10645 raft_consensus.cc:383] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:27.789942 10645 raft_consensus.cc:738] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:27.790196 10645 consensus_queue.cc:260] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.790784 10645 ts_tablet_manager.cc:1428] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250831 01:55:27.792073 10646 consensus_queue.cc:260] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.795307 10646 ts_tablet_manager.cc:1428] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250831 01:55:27.809278 9813 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]
I20250831 01:55:27.812777 9873 catalog_manager.cc:2747] Deleting table default.table_b436ecb230034afea5e7f8d551228511 [id=99d0e7e730954cc6abb603bb85b1db18] in response to Hive Metastore notification log event 22
W20250831 01:55:27.813115 9873 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 22 DROP_TABLE default.table_b436ecb230034afea5e7f8d551228511: Not found: the table does not exist: table_id: "99d0e7e730954cc6abb603bb85b1db18" table_name: "default.table_b436ecb230034afea5e7f8d551228511"
I20250831 01:55:27.813994 9873 catalog_manager.cc:2747] Deleting table default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a] in response to Hive Metastore notification log event 23
I20250831 01:55:27.825369 9873 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Sending DeleteTablet for 3 replicas of tablet d596b92ea9d745098c779e349a551e82
I20250831 01:55:27.826169 9984 tablet_service.cc:1515] Processing DeleteTablet for tablet d596b92ea9d745098c779e349a551e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:38384
I20250831 01:55:27.826510 10258 tablet_service.cc:1515] Processing DeleteTablet for tablet d596b92ea9d745098c779e349a551e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:48122
I20250831 01:55:27.826761 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet d596b92ea9d745098c779e349a551e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:42120
I20250831 01:55:27.826817 10623 tablet_replica.cc:331] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:27.827018 10621 tablet_replica.cc:331] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:27.827064 10623 raft_consensus.cc:2241] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Raft consensus shutting down.
I20250831 01:55:27.827229 10621 raft_consensus.cc:2241] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.827414 10621 raft_consensus.cc:2270] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.827646 10623 raft_consensus.cc:2270] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.827955 10619 tablet_replica.cc:331] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:27.828195 10619 raft_consensus.cc:2241] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.828210 10621 ts_tablet_manager.cc:1905] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.828394 10619 raft_consensus.cc:2270] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.828572 10623 ts_tablet_manager.cc:1905] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.834404 10621 ts_tablet_manager.cc:1918] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:27.834563 10621 log.cc:1199] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/d596b92ea9d745098c779e349a551e82
I20250831 01:55:27.835021 10621 ts_tablet_manager.cc:1939] T d596b92ea9d745098c779e349a551e82 P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:27.836231 9817 catalog_manager.cc:5674] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439) does not have the latest schema for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]). Expected version 1 got 0
I20250831 01:55:27.836469 9817 catalog_manager.cc:5674] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439) does not have the latest schema for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]). Expected version 1 got 0
I20250831 01:55:27.836751 10623 ts_tablet_manager.cc:1918] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:27.836782 10619 ts_tablet_manager.cc:1905] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:27.836974 10623 log.cc:1199] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/d596b92ea9d745098c779e349a551e82
I20250831 01:55:27.837455 10623 ts_tablet_manager.cc:1939] T d596b92ea9d745098c779e349a551e82 P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
W20250831 01:55:27.839033 9810 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a91e9d30219f48539efe3ebbbf819119 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 01:55:27.839437 9803 catalog_manager.cc:4928] TS 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): tablet d596b92ea9d745098c779e349a551e82 (table default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]) successfully deleted
W20250831 01:55:27.841554 9814 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_94d522ff92ac4ee7ae3906c381bfc49a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 01:55:27.841873 9809 catalog_manager.cc:5674] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085) does not have the latest schema for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]). Expected version 1 got 0
I20250831 01:55:27.842051 9801 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet d596b92ea9d745098c779e349a551e82 (table default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]) successfully deleted
I20250831 01:55:27.842124 9809 catalog_manager.cc:5674] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085) does not have the latest schema for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]). Expected version 1 got 0
I20250831 01:55:27.843464 9816 catalog_manager.cc:5674] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173) does not have the latest schema for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]). Expected version 1 got 0
I20250831 01:55:27.843485 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet d596b92ea9d745098c779e349a551e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-31 01:55:27 UTC) from {username='slave'} at 127.0.0.1:42120
I20250831 01:55:27.843637 10619 ts_tablet_manager.cc:1918] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:27.843786 10619 log.cc:1199] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/d596b92ea9d745098c779e349a551e82
I20250831 01:55:27.843672 9816 catalog_manager.cc:5674] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173) does not have the latest schema for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]). Expected version 1 got 0
I20250831 01:55:27.844261 10619 ts_tablet_manager.cc:1939] T d596b92ea9d745098c779e349a551e82 P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
W20250831 01:55:27.844738 9816 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c2b779e787cc4e808f7b74558e733471 consensus state has no leader
I20250831 01:55:27.845157 9816 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34126:
name: "default.Table_11af32d34145411d9cddea34ecfb51de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:27.845376 9802 catalog_manager.cc:4928] TS 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): tablet d596b92ea9d745098c779e349a551e82 (table default.table_519e74debc8b417c8fe427b8b4c3bd29 [id=b8a4751b36dc49a6906d7734ef84069a]) successfully deleted
W20250831 01:55:27.845728 9809 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0245596193a346d3aeb2d0aee88a6103 consensus state has no leader
W20250831 01:55:27.846094 9801 catalog_manager.cc:4895] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): delete failed for tablet d596b92ea9d745098c779e349a551e82 because the tablet was not found. No further retry: Not found: Tablet not found: d596b92ea9d745098c779e349a551e82
I20250831 01:55:27.846278 9809 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_3bbd369a7fa546ba9996f1935a35e1cd" } new_table_name: "default.Table_d32a6a9a414a4a9593f6e87746a6193a" modify_external_catalogs: true
I20250831 01:55:27.846952 9817 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34126:
name: "default.Table_76700aef3a6e4f2abae81047ab54898d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:27.854255 9669 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=5dc2dd483179499eafb15a91231b58ce]
I20250831 01:55:27.854669 9669 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 24) took 2803 ms. Trace:
I20250831 01:55:27.854774 9669 rpcz_store.cc:276] 0831 01:55:25.050929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:25.259875 (+208946us) service_pool.cc:225] Handling call
0831 01:55:26.174096 (+914221us) catalog_manager.cc:2076] Acquired catalog manager lock
0831 01:55:26.174629 (+ 533us) catalog_manager.cc:2122] Created new table and tablet info
0831 01:55:27.287658 (+1113029us) catalog_manager.cc:2148] Created new table in HMS catalog
0831 01:55:27.288470 (+ 812us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.288474 (+ 4us) write_op.cc:454] Released schema lock
0831 01:55:27.288706 (+ 232us) catalog_manager.cc:2156] Rolling back HMS table creation
0831 01:55:27.854596 (+565890us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 01:55:27.288390 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER)
0831 01:55:27.288402 (+ 12us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.288410 (+ 8us) write_op.cc:454] Released schema lock
0831 01:55:27.288415 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1449095,"Hive Metastore.run_cpu_time_us":525,"Hive Metastore.run_wall_time_us":228874,"child_traces":[["op",{}]]}
I20250831 01:55:27.885892 9812 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]
I20250831 01:55:27.889684 9668 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=2f6da2a74ea0463397b0da181cd0b2c1]
I20250831 01:55:27.890111 9668 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38174 (request call id 25) took 2812 ms. Trace:
I20250831 01:55:27.890240 9668 rpcz_store.cc:276] 0831 01:55:25.077388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:25.431679 (+354291us) service_pool.cc:225] Handling call
0831 01:55:26.175338 (+743659us) catalog_manager.cc:2076] Acquired catalog manager lock
0831 01:55:26.175788 (+ 450us) catalog_manager.cc:2122] Created new table and tablet info
0831 01:55:27.447535 (+1271747us) catalog_manager.cc:2148] Created new table in HMS catalog
0831 01:55:27.448362 (+ 827us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.448365 (+ 3us) write_op.cc:454] Released schema lock
0831 01:55:27.448605 (+ 240us) catalog_manager.cc:2156] Rolling back HMS table creation
0831 01:55:27.890061 (+441456us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 01:55:27.448284 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0b3d4b29236443d8a14d363ec2aab7c8 is not leader of this config: current role FOLLOWER)
0831 01:55:27.448297 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
0831 01:55:27.448304 (+ 7us) write_op.cc:454] Released schema lock
0831 01:55:27.448308 (+ 4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1517035,"Hive Metastore.run_cpu_time_us":527,"Hive Metastore.run_wall_time_us":195230,"child_traces":[["op",{}]]}
W20250831 01:55:27.896791 9803 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c2b779e787cc4e808f7b74558e733471 consensus state has no leader
I20250831 01:55:27.925318 9677 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]
I20250831 01:55:27.926364 9677 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:38174 (request call id 26) took 2832 ms. Trace:
I20250831 01:55:27.926482 9677 rpcz_store.cc:276] 0831 01:55:25.094273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 01:55:26.242307 (+1148034us) service_pool.cc:225] Handling call
0831 01:55:27.490190 (+1247883us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 01:55:27.926294 (+436104us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":399102,"Hive Metastore.run_cpu_time_us":226,"Hive Metastore.run_wall_time_us":35728}
I20250831 01:55:27.928378 9811 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]
F20250831 01:55:27.931802 9873 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 27, previous 27; dumping first 5 out of 5 received events: NotificationEvent(eventId=24, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_71f3e7f55e21496389aaf63260dec6a0, message={"server":"","servicePrincipal":"","db":"default","table":"table_71f3e7f55e21496389aaf63260dec6a0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_71f3e7f55e21496389aaf63260dec6a0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_71f3e7f55e21496389aaf63260dec6a0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_71f3e7f55e21496389aaf63260dec6a0\",\"kudu.table_id\":\"5dc2dd483179499eafb15a91231b58ce\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_54383364d8c14085a4963804ac2d6ba0, message={"server":"","servicePrincipal":"","db":"default","table":"table_54383364d8c14085a4963804ac2d6ba0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_54383364d8c14085a4963804ac2d6ba0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_54383364d8c14085a4963804ac2d6ba0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_54383364d8c14085a4963804ac2d6ba0\",\"kudu.table_id\":\"6b6b5b6d016b43f0921a4b22fe2aadd5\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_e5d7d8712d2d4ec181665ef359124956, message={"server":"","servicePrincipal":"","db":"default","table":"table_e5d7d8712d2d4ec181665ef359124956","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e5d7d8712d2d4ec181665ef359124956\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_e5d7d8712d2d4ec181665ef359124956\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e5d7d8712d2d4ec181665ef359124956\",\"kudu.table_id\":\"2f6da2a74ea0463397b0da181cd0b2c1\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_2faa27c5430c42e7b69a131d36238fdd, message={"server":"","servicePrincipal":"","db":"default","table":"table_2faa27c5430c42e7b69a131d36238fdd","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2faa27c5430c42e7b69a131d36238fdd\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_2faa27c5430c42e7b69a131d36238fdd\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2faa27c5430c42e7b69a131d36238fdd\",\"kudu.table_id\":\"9aa94d63ca7a4d22a2aefa09bbcf88cb\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_b0242810ea7a4fe3a1d3d8431eca51d2, message={"server":"","servicePrincipal":"","db":"default","table":"table_b0242810ea7a4fe3a1d3d8431eca51d2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b0242810ea7a4fe3a1d3d8431eca51d2\",\"kudu.table_id\":\"3e26e3f8edc04fa7a7ed3614967fb1aa\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f44d7cfedcd google::LogMessage::Fail() at ??:0
@ 0x7f44d7d02b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f44d7cfe7cc google::LogMessage::Flush() at ??:0
@ 0x7f44d7cfea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f44ee133794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250831 01:55:27.959422 9810 catalog_manager.cc:2149] created HMS catalog entry for table [id=6ca5ff2706e34ace8c9b111ac076142d]
@ 0x7f44ee132041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250831 01:55:27.961993 10647 raft_consensus.cc:491] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:27.962198 10647 raft_consensus.cc:513] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.963071 10647 leader_election.cc:290] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:27.963779 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0245596193a346d3aeb2d0aee88a6103" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:27.964016 10006 raft_consensus.cc:2466] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903e5d68fe5e4988a6b54623da741af5 in term 0.
I20250831 01:55:27.964476 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0245596193a346d3aeb2d0aee88a6103" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:27.964687 10146 raft_consensus.cc:2466] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903e5d68fe5e4988a6b54623da741af5 in term 0.
I20250831 01:55:27.965190 10213 leader_election.cc:304] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:27.965499 10647 raft_consensus.cc:2802] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:27.965621 10647 raft_consensus.cc:491] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:27.965715 10647 raft_consensus.cc:3058] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:27.969331 9810 master_service.cc:948] ReplaceTablet: received request to replace tablet 3772a8213fd74d1a995e0c8eed42c08b from {username='slave'} at 127.0.0.1:34126
I20250831 01:55:27.969584 10647 raft_consensus.cc:513] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.970654 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0245596193a346d3aeb2d0aee88a6103" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:27.970862 10146 raft_consensus.cc:3058] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:27.973574 10146 raft_consensus.cc:2466] T 0245596193a346d3aeb2d0aee88a6103 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903e5d68fe5e4988a6b54623da741af5 in term 1.
I20250831 01:55:27.974062 10213 leader_election.cc:304] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:27.974543 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0245596193a346d3aeb2d0aee88a6103" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:27.974767 10006 raft_consensus.cc:3058] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:27.977545 10006 raft_consensus.cc:2466] T 0245596193a346d3aeb2d0aee88a6103 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903e5d68fe5e4988a6b54623da741af5 in term 1.
I20250831 01:55:27.978168 10647 leader_election.cc:290] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 election: Requested vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:27.978410 10647 raft_consensus.cc:2802] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:27.978549 10647 raft_consensus.cc:695] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [term 1 LEADER]: Becoming Leader. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Running, Role: LEADER
I20250831 01:55:27.979669 10647 consensus_queue.cc:237] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:27.981611 10258 tablet_service.cc:1468] Processing CreateTablet for tablet 2e4ccddb53c642bc83e958032cc94c62 (DEFAULT_TABLE table=default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.981709 9984 tablet_service.cc:1468] Processing CreateTablet for tablet 2e4ccddb53c642bc83e958032cc94c62 (DEFAULT_TABLE table=default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:27.982092 10258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4ccddb53c642bc83e958032cc94c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:27.982087 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4ccddb53c642bc83e958032cc94c62. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f44ee139cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250831 01:55:27.988766 10645 tablet_bootstrap.cc:492] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:27.991151 10645 tablet_bootstrap.cc:654] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:27.991178 9810 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Sending DeleteTablet for 3 replicas of tablet 3772a8213fd74d1a995e0c8eed42c08b
I20250831 01:55:27.991756 9810 catalog_manager.cc:6652] ReplaceTablet: tablet 3772a8213fd74d1a995e0c8eed42c08b deleted and replaced by tablet cd278fc485cb44798d88871b367994fc
I20250831 01:55:27.993049 9810 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34126:
table { table_name: "default.Table_a3dfa51fdb0a42d390d8b0c78c181087" } new_table_name: "default.Table_f6625a8fb0a64ad5985877617c5a9e18" modify_external_catalogs: true
@ 0x7f44ee139b39 std::_Function_handler<>::_M_invoke() at ??:0
I20250831 01:55:27.997501 9987 tablet_service.cc:1515] Processing DeleteTablet for tablet 3772a8213fd74d1a995e0c8eed42c08b with delete_type TABLET_DATA_DELETED (replaced by tablet cd278fc485cb44798d88871b367994fc) from {username='slave'} at 127.0.0.1:38384
@ 0xa774dd std::function<>::operator()() at ??:?
I20250831 01:55:27.997840 10623 tablet_replica.cc:331] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:27.998034 10623 raft_consensus.cc:2241] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:27.998275 10623 raft_consensus.cc:2270] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:27.998903 10623 ts_tablet_manager.cc:1905] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:28.001708 10645 tablet_bootstrap.cc:492] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:28.001925 10645 ts_tablet_manager.cc:1397] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
@ 0x7f44daa5bcc9 kudu::Thread::SuperviseThread() at ??:0
I20250831 01:55:28.002949 10645 raft_consensus.cc:357] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.003168 10645 raft_consensus.cc:383] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:28.003259 10645 raft_consensus.cc:738] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:28.003513 10645 consensus_queue.cc:260] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.004163 10645 ts_tablet_manager.cc:1428] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
@ 0x7f44dde7d6db start_thread at ??:0
I20250831 01:55:28.008790 10259 tablet_service.cc:1468] Processing CreateTablet for tablet cd278fc485cb44798d88871b367994fc (DEFAULT_TABLE table=default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:28.009279 10259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd278fc485cb44798d88871b367994fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:28.014834 9987 tablet_service.cc:1468] Processing CreateTablet for tablet cd278fc485cb44798d88871b367994fc (DEFAULT_TABLE table=default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:28.015358 9987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd278fc485cb44798d88871b367994fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:28.024013 10643 tablet_bootstrap.cc:492] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
@ 0x7f44d6baf71f clone at ??:0
I20250831 01:55:28.025940 10643 tablet_bootstrap.cc:654] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:28.034116 10645 tablet_bootstrap.cc:492] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:28.035883 10645 tablet_bootstrap.cc:654] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:28.036967 10258 tablet_service.cc:1515] Processing DeleteTablet for tablet 3772a8213fd74d1a995e0c8eed42c08b with delete_type TABLET_DATA_DELETED (replaced by tablet cd278fc485cb44798d88871b367994fc) from {username='slave'} at 127.0.0.1:48122
I20250831 01:55:28.037338 10619 tablet_replica.cc:331] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:28.037564 10619 raft_consensus.cc:2241] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:28.037825 10619 raft_consensus.cc:2270] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:28.038113 10645 tablet_bootstrap.cc:492] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:28.038261 10645 ts_tablet_manager.cc:1397] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250831 01:55:28.038362 10619 ts_tablet_manager.cc:1905] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:28.038762 10623 ts_tablet_manager.cc:1918] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:28.038933 10623 log.cc:1199] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/3772a8213fd74d1a995e0c8eed42c08b
I20250831 01:55:28.039160 10645 raft_consensus.cc:357] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.039402 10645 raft_consensus.cc:383] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:28.039462 10623 ts_tablet_manager.cc:1939] T 3772a8213fd74d1a995e0c8eed42c08b P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:28.039479 10645 raft_consensus.cc:738] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:28.039750 10645 consensus_queue.cc:260] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.040253 10645 ts_tablet_manager.cc:1428] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250831 01:55:28.043387 10643 tablet_bootstrap.cc:492] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:28.043545 10643 ts_tablet_manager.cc:1397] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.009s
I20250831 01:55:28.044169 10643 raft_consensus.cc:357] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.044375 10643 raft_consensus.cc:383] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:28.044471 10643 raft_consensus.cc:738] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:28.044705 10643 consensus_queue.cc:260] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.046546 10619 ts_tablet_manager.cc:1918] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:28.048710 10643 ts_tablet_manager.cc:1428] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250831 01:55:28.049549 10643 tablet_bootstrap.cc:492] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:28.049873 10619 log.cc:1199] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/3772a8213fd74d1a995e0c8eed42c08b
I20250831 01:55:28.050345 10619 ts_tablet_manager.cc:1939] T 3772a8213fd74d1a995e0c8eed42c08b P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:28.052160 10643 tablet_bootstrap.cc:654] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:28.054198 10643 tablet_bootstrap.cc:492] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:28.054353 10643 ts_tablet_manager.cc:1397] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250831 01:55:28.055087 10643 raft_consensus.cc:357] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.055274 10643 raft_consensus.cc:383] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:28.055363 10643 raft_consensus.cc:738] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:28.055609 10643 consensus_queue.cc:260] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.056806 10643 ts_tablet_manager.cc:1428] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250831 01:55:28.088057 10462 raft_consensus.cc:491] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:28.088263 10462 raft_consensus.cc:513] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:28.088882 10462 leader_election.cc:290] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:28.089517 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2b779e787cc4e808f7b74558e733471" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:28.089567 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2b779e787cc4e808f7b74558e733471" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:28.089700 10279 raft_consensus.cc:2466] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:28.089831 10146 raft_consensus.cc:2466] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:28.090216 9942 leader_election.cc:304] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters:
I20250831 01:55:28.090520 10462 raft_consensus.cc:2802] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:28.090608 10462 raft_consensus.cc:491] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:28.090690 10462 raft_consensus.cc:3058] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:28.093384 10462 raft_consensus.cc:513] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:28.094686 10462 leader_election.cc:290] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:28.094833 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2b779e787cc4e808f7b74558e733471" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:28.095067 10279 raft_consensus.cc:3058] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:28.095096 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2b779e787cc4e808f7b74558e733471" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:28.095319 10146 raft_consensus.cc:3058] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:28.097177 10146 raft_consensus.cc:2466] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:28.097723 9941 leader_election.cc:304] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:28.098102 10462 raft_consensus.cc:2802] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:28.098100 10279 raft_consensus.cc:2466] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:28.098265 10462 raft_consensus.cc:695] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:28.098565 10462 consensus_queue.cc:237] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } }
I20250831 01:55:28.128592 10462 raft_consensus.cc:491] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:28.128796 10462 raft_consensus.cc:513] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.129436 10462 leader_election.cc:290] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:28.130007 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd278fc485cb44798d88871b367994fc" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:28.130251 10279 raft_consensus.cc:2466] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
W20250831 01:55:28.130507 9941 leader_election.cc:343] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Not found: Tablet not found: cd278fc485cb44798d88871b367994fc
I20250831 01:55:28.130722 9942 leader_election.cc:304] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters: fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:28.131093 10462 raft_consensus.cc:2802] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:28.131239 10462 raft_consensus.cc:491] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:28.131331 10462 raft_consensus.cc:3058] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:28.133278 10462 raft_consensus.cc:513] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.133905 10462 leader_election.cc:290] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:28.134426 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd278fc485cb44798d88871b367994fc" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:28.134650 10279 raft_consensus.cc:3058] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
W20250831 01:55:28.134778 9941 leader_election.cc:343] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Not found: Tablet not found: cd278fc485cb44798d88871b367994fc
I20250831 01:55:28.136631 10279 raft_consensus.cc:2466] T cd278fc485cb44798d88871b367994fc P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:28.137068 9942 leader_election.cc:304] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters: fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:28.137343 10462 raft_consensus.cc:2802] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:28.137492 10462 raft_consensus.cc:695] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:28.137828 10462 consensus_queue.cc:237] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
W20250831 01:55:28.268574 10214 connection.cc:537] server connection from 127.0.0.1:48122 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:28.268579 9944 connection.cc:537] server connection from 127.0.0.1:38384 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:28.268930 10214 connection.cc:537] client connection to 127.9.47.124:37481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:28.269158 10335 connection.cc:537] client connection to 127.9.47.124:37481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:28.270395 9942 connection.cc:537] client connection to 127.9.47.124:37481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250831 01:55:28.269831 10422 master_runner.cc:186] Error getting master registration for 127.9.47.124:37481: Network error: recv got EOF from 127.9.47.124:37481 (error 108),
I20250831 01:55:28.274798 10422 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:55:28.300958 10080 consensus_peers.cc:489] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 -> Peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Couldn't send request to peer 903e5d68fe5e4988a6b54623da741af5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3772a8213fd74d1a995e0c8eed42c08b. This is attempt 1: this message will repeat every 5th retry.
W20250831 01:55:28.323153 10081 consensus_peers.cc:489] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 -> Peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): Couldn't send request to peer 67e9e49e7a864057adb3e685607c9844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3772a8213fd74d1a995e0c8eed42c08b. This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:28.330142 10462 raft_consensus.cc:491] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:28.330367 10462 raft_consensus.cc:513] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.330962 10462 leader_election.cc:290] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:28.331513 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e4ccddb53c642bc83e958032cc94c62" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:28.331751 10279 raft_consensus.cc:2466] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
W20250831 01:55:28.331781 9941 leader_election.cc:343] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Not found: Tablet not found: 2e4ccddb53c642bc83e958032cc94c62
I20250831 01:55:28.332123 9942 leader_election.cc:304] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters: fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:28.332386 10462 raft_consensus.cc:2802] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:28.332496 10462 raft_consensus.cc:491] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:28.332602 10462 raft_consensus.cc:3058] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:28.334342 10462 raft_consensus.cc:513] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.334872 10462 leader_election.cc:290] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085), 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:28.335362 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e4ccddb53c642bc83e958032cc94c62" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:28.335543 10279 raft_consensus.cc:3058] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
W20250831 01:55:28.335665 9941 leader_election.cc:343] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Not found: Tablet not found: 2e4ccddb53c642bc83e958032cc94c62
I20250831 01:55:28.337306 10279 raft_consensus.cc:2466] T 2e4ccddb53c642bc83e958032cc94c62 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:28.337682 9942 leader_election.cc:304] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters: fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:28.337970 10462 raft_consensus.cc:2802] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:28.338140 10462 raft_consensus.cc:695] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:28.338426 10462 consensus_queue.cc:237] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:28.476418 10682 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:28.476603 10682 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:28.477190 10682 leader_election.cc:290] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:28.477591 10535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 59 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" is_pre_election: true
W20250831 01:55:28.479065 9662 leader_election.cc:336] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:28.479179 9662 leader_election.cc:304] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8; no voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:28.479405 10682 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 01:55:28.519372 10685 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:28.519661 10685 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:28.521323 10685 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:28.522050 9686 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" candidate_term: 3 candidate_status { last_received { term: 2 index: 61 } } ignore_live_leader: false dest_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" is_pre_election: true
I20250831 01:55:28.522370 9686 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b3cdeb22a7546b988ab9938f8dc1027 in term 2.
W20250831 01:55:28.522532 10489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:28.523079 10488 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027; no voters:
I20250831 01:55:28.523556 10685 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 01:55:28.523718 10685 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:28.523849 10685 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 2 FOLLOWER]: Advancing to term 3
W20250831 01:55:28.524526 10489 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:28.526448 10685 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:28.527210 10685 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 3 election: Requested vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:28.527963 9686 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" candidate_term: 3 candidate_status { last_received { term: 2 index: 61 } } ignore_live_leader: false dest_uuid: "0b3d4b29236443d8a14d363ec2aab7c8"
I20250831 01:55:28.528163 9686 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 2 FOLLOWER]: Advancing to term 3
W20250831 01:55:28.528559 10489 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:28.530416 9686 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b3cdeb22a7546b988ab9938f8dc1027 in term 3.
I20250831 01:55:28.530802 10488 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:28.531123 10685 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 01:55:28.531515 10685 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 LEADER]: Becoming Leader. State: Replica: 6b3cdeb22a7546b988ab9938f8dc1027, State: Running, Role: LEADER
I20250831 01:55:28.531940 10685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 2.61, Last appended by leader: 61, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:28.533245 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3cdeb22a7546b988ab9938f8dc1027. Latest consensus state: current_term: 3 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:28.533496 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:28.534085 10689 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 01:55:28.536773 10689 catalog_manager.cc:671] Loaded metadata for table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]
I20250831 01:55:28.537312 10689 catalog_manager.cc:671] Loaded metadata for table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]
I20250831 01:55:28.538422 10689 catalog_manager.cc:671] Loaded metadata for table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d]
I20250831 01:55:28.538915 10689 catalog_manager.cc:671] Loaded metadata for table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]
I20250831 01:55:28.539366 10689 catalog_manager.cc:671] Loaded metadata for table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]
I20250831 01:55:28.539726 10689 catalog_manager.cc:671] Loaded metadata for table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]
I20250831 01:55:28.540053 10689 catalog_manager.cc:671] Loaded metadata for table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]
I20250831 01:55:28.540570 10689 catalog_manager.cc:671] Loaded metadata for table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7]
I20250831 01:55:28.541043 10689 catalog_manager.cc:671] Loaded metadata for table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]
I20250831 01:55:28.543653 10689 tablet_loader.cc:96] loaded metadata for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178])
I20250831 01:55:28.544394 10689 tablet_loader.cc:96] loaded metadata for tablet 0fd06ceda8134e459d5bb86fac5cdbb8 (table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb])
I20250831 01:55:28.544889 10689 tablet_loader.cc:96] loaded metadata for tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 (table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b])
I20250831 01:55:28.545182 10689 tablet_loader.cc:96] loaded metadata for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7])
I20250831 01:55:28.545500 10689 tablet_loader.cc:96] loaded metadata for tablet 2e4ccddb53c642bc83e958032cc94c62 (table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d])
I20250831 01:55:28.546018 10689 tablet_loader.cc:96] loaded metadata for tablet 4ee86091bc604b6a9d33c56d091fbd4d (table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5])
I20250831 01:55:28.546785 10689 tablet_loader.cc:96] loaded metadata for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792])
I20250831 01:55:28.547072 10689 tablet_loader.cc:96] loaded metadata for tablet cd278fc485cb44798d88871b367994fc (table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194])
I20250831 01:55:28.547372 10689 tablet_loader.cc:96] loaded metadata for tablet d36d4612593e4d8985b160918d54f87a (table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa])
I20250831 01:55:28.547938 10689 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 01:55:28.549311 10689 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:28.549405 10689 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 01:55:28.551363 10689 catalog_manager.cc:1506] Loading token signing keys...
I20250831 01:55:28.552403 10689 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Loaded TSK: 0
I20250831 01:55:28.553299 10689 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 01:55:28.554409 10689 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250831 01:55:28.555454 10689 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 23
I20250831 01:55:28.575318 10525 master_service.cc:948] ReplaceTablet: received request to replace tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 from {username='slave'} at 127.0.0.1:41902
I20250831 01:55:28.575680 10524 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41902:
name: "default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:28.575985 10520 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41902:
name: "default.Table_18f5062db5284a5d9fa9ecc471dc9af7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:28.576582 10517 master_service.cc:948] ReplaceTablet: received request to replace tablet 0fd06ceda8134e459d5bb86fac5cdbb8 from {username='slave'} at 127.0.0.1:41902
I20250831 01:55:28.576962 10503 master_service.cc:948] ReplaceTablet: received request to replace tablet d36d4612593e4d8985b160918d54f87a from {username='slave'} at 127.0.0.1:41902
I20250831 01:55:28.577287 10514 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ee86091bc604b6a9d33c56d091fbd4d from {username='slave'} at 127.0.0.1:41902
I20250831 01:55:28.579584 10523 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41902:
name: "default.Table_e5d7d8712d2d4ec181665ef359124956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:28.580937 10521 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41902:
name: "default.Table_b436ecb230034afea5e7f8d551228511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:28.581981 10522 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41902:
name: "default.Table_71f3e7f55e21496389aaf63260dec6a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250831 01:55:28.582697 10602 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 27, previous 27; dumping first 5 out of 7 received events: NotificationEvent(eventId=24, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_71f3e7f55e21496389aaf63260dec6a0, message={"server":"","servicePrincipal":"","db":"default","table":"table_71f3e7f55e21496389aaf63260dec6a0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_71f3e7f55e21496389aaf63260dec6a0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_71f3e7f55e21496389aaf63260dec6a0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_71f3e7f55e21496389aaf63260dec6a0\",\"kudu.table_id\":\"5dc2dd483179499eafb15a91231b58ce\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_54383364d8c14085a4963804ac2d6ba0, message={"server":"","servicePrincipal":"","db":"default","table":"table_54383364d8c14085a4963804ac2d6ba0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_54383364d8c14085a4963804ac2d6ba0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_54383364d8c14085a4963804ac2d6ba0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_54383364d8c14085a4963804ac2d6ba0\",\"kudu.table_id\":\"6b6b5b6d016b43f0921a4b22fe2aadd5\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_e5d7d8712d2d4ec181665ef359124956, message={"server":"","servicePrincipal":"","db":"default","table":"table_e5d7d8712d2d4ec181665ef359124956","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e5d7d8712d2d4ec181665ef359124956\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_e5d7d8712d2d4ec181665ef359124956\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e5d7d8712d2d4ec181665ef359124956\",\"kudu.table_id\":\"2f6da2a74ea0463397b0da181cd0b2c1\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_2faa27c5430c42e7b69a131d36238fdd, message={"server":"","servicePrincipal":"","db":"default","table":"table_2faa27c5430c42e7b69a131d36238fdd","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2faa27c5430c42e7b69a131d36238fdd\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_2faa27c5430c42e7b69a131d36238fdd\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2faa27c5430c42e7b69a131d36238fdd\",\"kudu.table_id\":\"9aa94d63ca7a4d22a2aefa09bbcf88cb\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_b0242810ea7a4fe3a1d3d8431eca51d2, message={"server":"","servicePrincipal":"","db":"default","table":"table_b0242810ea7a4fe3a1d3d8431eca51d2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b0242810ea7a4fe3a1d3d8431eca51d2\",\"kudu.table_id\":\"3e26e3f8edc04fa7a7ed3614967fb1aa\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250831 01:55:28.582676 10518 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41902:
table { table_name: "default.Table_a91e9d30219f48539efe3ebbbf819119" } modify_external_catalogs: true
I20250831 01:55:28.583379 10518 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41902:
table { table_name: "default.Table_a91e9d30219f48539efe3ebbbf819119" } modify_external_catalogs: true
@ 0x7f67f68c0dcd google::LogMessage::Fail() at ??:0
W20250831 01:55:28.586702 10489 consensus_peers.cc:489] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 -> Peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Couldn't send request to peer 68251650a6ad4c1eba8826b2949c8905. Status: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f67f68c4b93 google::LogMessage::SendToLog() at ??:0
I20250831 01:55:28.587293 9686 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Refusing update from remote peer 6b3cdeb22a7546b988ab9938f8dc1027: Log matching property violated. Preceding OpId in replica: term: 2 index: 59. Preceding OpId from leader: term: 3 index: 64. (index mismatch)
I20250831 01:55:28.588023 10691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 58, Time since last communication: 0.000s
@ 0x7f67f68c07cc google::LogMessage::Flush() at ??:0
@ 0x7f67f68c0a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f680ccf5794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250831 01:55:28.595634 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3cdeb22a7546b988ab9938f8dc1027. Latest consensus state: current_term: 3 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:28.595914 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f680ccf4041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250831 01:55:28.601518 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:28.601835 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f680ccfbcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250831 01:55:28.606132 10525 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Sending DeleteTablet for 3 replicas of tablet 1def9b41fbe94b4ba680d0d4afd0d2b8
I20250831 01:55:28.606608 10517 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Sending DeleteTablet for 3 replicas of tablet 0fd06ceda8134e459d5bb86fac5cdbb8
I20250831 01:55:28.607826 10517 catalog_manager.cc:6652] ReplaceTablet: tablet 0fd06ceda8134e459d5bb86fac5cdbb8 deleted and replaced by tablet 970335514c734553b2360365c81e2d51
I20250831 01:55:28.608280 10517 master_service.cc:948] ReplaceTablet: received request to replace tablet c2b779e787cc4e808f7b74558e733471 from {username='slave'} at 127.0.0.1:41902
I20250831 01:55:28.608418 10525 catalog_manager.cc:6652] ReplaceTablet: tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 deleted and replaced by tablet d3303431c6684d609b9238d908ad3cf4
@ 0x7f680ccfbb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f67f961dcc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f67fca3f6db start_thread at ??:0
@ 0x7f67f577171f clone at ??:0
W20250831 01:55:28.810484 10336 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.47.125:46791)
W20250831 01:55:28.810710 10702 negotiation.cc:337] Failed RPC negotiation. Trace:
0831 01:55:28.609082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.9.47.67:33173)
0831 01:55:28.630351 (+ 21269us) server_negotiation.cc:206] Beginning negotiation
0831 01:55:28.630361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0831 01:55:28.809848 (+179487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48128: BlockingRecv error: recv got EOF from 127.0.0.1:48128 (error 108)
Metrics: {"server-negotiator.queue_time_us":21111,"thread_start_us":21280,"threads_started":1}
W20250831 01:55:28.810740 10704 negotiation.cc:337] Failed RPC negotiation. Trace:
0831 01:55:28.610652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.9.47.66:46085)
0831 01:55:28.630332 (+ 19680us) server_negotiation.cc:206] Beginning negotiation
0831 01:55:28.630344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0831 01:55:28.809824 (+179480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42126: BlockingRecv error: recv got EOF from 127.0.0.1:42126 (error 108)
Metrics: {"server-negotiator.queue_time_us":19493,"thread_start_us":19701,"threads_started":1}
W20250831 01:55:28.810750 10336 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.47.125:46791)
W20250831 01:55:28.811193 10336 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.47.125:46791)
W20250831 01:55:28.812036 10336 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.47.125:46791)
W20250831 01:55:28.812469 10701 negotiation.cc:337] Failed RPC negotiation. Trace:
0831 01:55:28.607731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.9.47.65:43439)
0831 01:55:28.630819 (+ 23088us) server_negotiation.cc:206] Beginning negotiation
0831 01:55:28.630828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0831 01:55:28.630853 (+ 25us) server_negotiation.cc:407] Connection header received
0831 01:55:28.630954 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0831 01:55:28.630965 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0831 01:55:28.631075 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0831 01:55:28.631224 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0831 01:55:28.811664 (+180440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38390: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":22953,"thread_start_us":22869,"threads_started":1}
W20250831 01:55:28.815183 10081 connection.cc:537] client connection to 127.9.47.125:46791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:28.816862 10681 negotiation.cc:337] Failed RPC negotiation. Trace:
0831 01:55:28.815865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.47.125:46791 (local address 127.9.47.66:42915)
0831 01:55:28.816069 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0831 01:55:28.816089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0831 01:55:28.816180 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0831 01:55:28.816610 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":59}
W20250831 01:55:28.817525 10678 negotiation.cc:337] Failed RPC negotiation. Trace:
0831 01:55:28.813065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.47.125:46791 (local address 127.0.0.1:41912)
0831 01:55:28.813276 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0831 01:55:28.813296 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0831 01:55:28.813429 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0831 01:55:28.816200 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":69}
I20250831 01:55:28.969369 10462 consensus_queue.cc:1035] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 01:55:28.976243 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:29.029377 10716 consensus_queue.cc:1035] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:29.035164 10724 consensus_queue.cc:1035] T 0245596193a346d3aeb2d0aee88a6103 P 903e5d68fe5e4988a6b54623da741af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:29.098941 10682 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b3cdeb22a7546b988ab9938f8dc1027)
I20250831 01:55:29.099128 10682 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:29.099767 10682 leader_election.cc:290] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:29.101696 9663 leader_election.cc:336] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:29.101933 9662 leader_election.cc:336] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:29.102057 9662 leader_election.cc:304] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8; no voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:29.102321 10682 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 01:55:29.152470 10718 consensus_queue.cc:1035] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 01:55:29.157332 9941 consensus_peers.cc:489] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e4ccddb53c642bc83e958032cc94c62. This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:29.183753 10733 consensus_queue.cc:1035] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:29.191011 10718 consensus_queue.cc:1035] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:29.714464 10745 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b3cdeb22a7546b988ab9938f8dc1027)
I20250831 01:55:29.714648 10745 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:29.716238 10745 leader_election.cc:290] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:29.717499 9663 leader_election.cc:336] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:29.717701 9662 leader_election.cc:336] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:29.717857 9662 leader_election.cc:304] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8; no voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:29.718107 10745 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 01:55:30.017341 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--webserver_interface=127.9.47.124
--webserver_port=42935
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250831 01:55:30.122953 10757 ts_tablet_manager.cc:927] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Initiating tablet copy from peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:30.124128 10757 tablet_copy_client.cc:323] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.47.65:43439
I20250831 01:55:30.125172 10028 tablet_copy_service.cc:140] P 67e9e49e7a864057adb3e685607c9844: Received BeginTabletCopySession request for tablet cd278fc485cb44798d88871b367994fc from peer fbe14365c42e4c6ba1173d829f997f81 ({username='slave'} at 127.9.47.66:36083)
I20250831 01:55:30.125339 10028 tablet_copy_service.cc:161] P 67e9e49e7a864057adb3e685607c9844: Beginning new tablet copy session on tablet cd278fc485cb44798d88871b367994fc from peer fbe14365c42e4c6ba1173d829f997f81 at {username='slave'} at 127.9.47.66:36083: session id = fbe14365c42e4c6ba1173d829f997f81-cd278fc485cb44798d88871b367994fc
I20250831 01:55:30.127779 10028 tablet_copy_source_session.cc:215] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 01:55:30.129573 10757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd278fc485cb44798d88871b367994fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:30.134096 10757 tablet_copy_client.cc:806] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Starting download of 0 data blocks...
I20250831 01:55:30.134420 10757 tablet_copy_client.cc:670] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Starting download of 1 WAL segments...
I20250831 01:55:30.136693 10757 tablet_copy_client.cc:538] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 01:55:30.139504 10757 tablet_bootstrap.cc:492] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:30.144620 10757 tablet_bootstrap.cc:492] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:30.145007 10757 tablet_bootstrap.cc:492] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Bootstrap complete.
I20250831 01:55:30.145232 10757 ts_tablet_manager.cc:1397] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20250831 01:55:30.145938 10757 raft_consensus.cc:357] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:30.146097 10757 raft_consensus.cc:738] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:30.146371 10757 consensus_queue.cc:260] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:30.147154 10757 ts_tablet_manager.cc:1428] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250831 01:55:30.148245 10028 tablet_copy_service.cc:342] P 67e9e49e7a864057adb3e685607c9844: Request end of tablet copy session fbe14365c42e4c6ba1173d829f997f81-cd278fc485cb44798d88871b367994fc received from {username='slave'} at 127.9.47.66:36083
I20250831 01:55:30.148422 10028 tablet_copy_service.cc:434] P 67e9e49e7a864057adb3e685607c9844: ending tablet copy session fbe14365c42e4c6ba1173d829f997f81-cd278fc485cb44798d88871b367994fc on tablet cd278fc485cb44798d88871b367994fc with peer fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:30.153745 10757 ts_tablet_manager.cc:927] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: Initiating tablet copy from peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:30.154546 10757 tablet_copy_client.cc:323] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.47.65:43439
I20250831 01:55:30.155148 10028 tablet_copy_service.cc:140] P 67e9e49e7a864057adb3e685607c9844: Received BeginTabletCopySession request for tablet 2e4ccddb53c642bc83e958032cc94c62 from peer fbe14365c42e4c6ba1173d829f997f81 ({username='slave'} at 127.9.47.66:36083)
I20250831 01:55:30.155303 10028 tablet_copy_service.cc:161] P 67e9e49e7a864057adb3e685607c9844: Beginning new tablet copy session on tablet 2e4ccddb53c642bc83e958032cc94c62 from peer fbe14365c42e4c6ba1173d829f997f81 at {username='slave'} at 127.9.47.66:36083: session id = fbe14365c42e4c6ba1173d829f997f81-2e4ccddb53c642bc83e958032cc94c62
I20250831 01:55:30.157114 10028 tablet_copy_source_session.cc:215] T 2e4ccddb53c642bc83e958032cc94c62 P 67e9e49e7a864057adb3e685607c9844: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 01:55:30.158299 10757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4ccddb53c642bc83e958032cc94c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:30.162431 10757 tablet_copy_client.cc:806] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Starting download of 0 data blocks...
I20250831 01:55:30.162659 10757 tablet_copy_client.cc:670] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Starting download of 1 WAL segments...
I20250831 01:55:30.164127 10757 tablet_copy_client.cc:538] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 01:55:30.166383 10757 tablet_bootstrap.cc:492] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:30.171640 10757 tablet_bootstrap.cc:492] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:30.172040 10757 tablet_bootstrap.cc:492] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap complete.
I20250831 01:55:30.172262 10757 ts_tablet_manager.cc:1397] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250831 01:55:30.172986 10757 raft_consensus.cc:357] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:30.173172 10757 raft_consensus.cc:738] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:30.173467 10757 consensus_queue.cc:260] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } } peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } }
I20250831 01:55:30.174129 10757 ts_tablet_manager.cc:1428] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250831 01:55:30.179446 10028 tablet_copy_service.cc:342] P 67e9e49e7a864057adb3e685607c9844: Request end of tablet copy session fbe14365c42e4c6ba1173d829f997f81-2e4ccddb53c642bc83e958032cc94c62 received from {username='slave'} at 127.9.47.66:36083
I20250831 01:55:30.179585 10028 tablet_copy_service.cc:434] P 67e9e49e7a864057adb3e685607c9844: ending tablet copy session fbe14365c42e4c6ba1173d829f997f81-2e4ccddb53c642bc83e958032cc94c62 on tablet 2e4ccddb53c642bc83e958032cc94c62 with peer fbe14365c42e4c6ba1173d829f997f81
W20250831 01:55:30.363235 10754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:30.363559 10754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:30.363669 10754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:30.371827 10754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:30.371942 10754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:30.372043 10754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:30.372109 10754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:30.382237 10754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.47.124
--webserver_port=42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:30.383844 10754 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:30.385620 10754 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:30.393637 10768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:30.393905 10771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:30.393823 10769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:30.394523 10754 server_base.cc:1047] running on GCE node
I20250831 01:55:30.395291 10754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:30.396670 10754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:30.397862 10754 hybrid_clock.cc:648] HybridClock initialized: now 1756605330397818 us; error 43 us; skew 500 ppm
I20250831 01:55:30.398191 10754 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:30.400362 10754 webserver.cc:480] Webserver started at http://127.9.47.124:42935/ using document root <none> and password file <none>
I20250831 01:55:30.401098 10754 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:30.401207 10754 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:30.407883 10754 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250831 01:55:30.410739 10777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:30.412138 10754 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250831 01:55:30.412300 10754 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:30.412972 10754 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:30.437254 10754 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:30.438334 10754 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:30.438674 10754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:30.461042 10754 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.124:37481
I20250831 01:55:30.461045 10828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.124:37481 every 8 connection(s)
I20250831 01:55:30.463061 10754 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
I20250831 01:55:30.466303 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 10754
I20250831 01:55:30.470007 10829 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:30.473623 10829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap starting.
I20250831 01:55:30.488245 10829 log.cc:826] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:30.540491 10829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=43 ignored=0} mutations{seen=47 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:30.541181 10829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap complete.
I20250831 01:55:30.543722 10053 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:30.547315 10829 raft_consensus.cc:357] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:30.547959 10829 raft_consensus.cc:738] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Initialized, Role: FOLLOWER
I20250831 01:55:30.548566 10829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 2.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:30.549288 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:30.549518 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:30.550340 10829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:30.566007 10849 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:30.566210 10849 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: loading cluster ID for follower catalog manager: success
I20250831 01:55:30.572188 10849 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: acquiring CA information for follower catalog manager: success
I20250831 01:55:30.575297 10849 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:30.581477 10754 master_runner.cc:186] Error getting master registration for 127.9.47.125:46791: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111),
I20250831 01:55:30.585462 10754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:30.696770 10325 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:30.699211 10794 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:58393; Asking this server to re-register.
I20250831 01:55:30.699961 10325 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:30.701730 10794 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:30.836545 10856 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b3cdeb22a7546b988ab9938f8dc1027)
I20250831 01:55:30.836755 10856 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:30.837515 10856 leader_election.cc:290] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:30.849642 9663 leader_election.cc:336] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:30.850325 10804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 66 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905" is_pre_election: true
I20250831 01:55:30.851054 10804 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b3d4b29236443d8a14d363ec2aab7c8 in term 2.
I20250831 01:55:30.851609 9662 leader_election.cc:304] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905; no voters: 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:30.851900 10856 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 01:55:30.852046 10856 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b3cdeb22a7546b988ab9938f8dc1027)
I20250831 01:55:30.852141 10856 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 3 FOLLOWER]: Advancing to term 4
I20250831 01:55:30.854079 10856 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:30.854715 10856 leader_election.cc:290] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 election: Requested vote from peers 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:30.855211 10804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 66 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905"
I20250831 01:55:30.855394 10804 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 2 FOLLOWER]: Advancing to term 4
W20250831 01:55:30.856381 9663 leader_election.cc:336] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:30.859536 10804 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b3d4b29236443d8a14d363ec2aab7c8 in term 4.
I20250831 01:55:30.859928 9662 leader_election.cc:304] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905; no voters: 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:30.860234 10856 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 01:55:30.860647 10856 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0b3d4b29236443d8a14d363ec2aab7c8, State: Running, Role: LEADER
I20250831 01:55:30.860931 10856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 3.66, Last appended by leader: 66, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:30.862341 10861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3d4b29236443d8a14d363ec2aab7c8. Latest consensus state: current_term: 4 leader_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:30.862625 10861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:30.863242 10863 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 01:55:30.865542 10863 catalog_manager.cc:671] Loaded metadata for table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]
I20250831 01:55:30.865984 10863 catalog_manager.cc:671] Loaded metadata for table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]
I20250831 01:55:30.866355 10863 catalog_manager.cc:671] Loaded metadata for table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d]
I20250831 01:55:30.866686 10863 catalog_manager.cc:671] Loaded metadata for table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]
I20250831 01:55:30.866991 10863 catalog_manager.cc:671] Loaded metadata for table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]
I20250831 01:55:30.867309 10863 catalog_manager.cc:671] Loaded metadata for table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]
I20250831 01:55:30.867601 10863 catalog_manager.cc:671] Loaded metadata for table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]
I20250831 01:55:30.868069 10863 catalog_manager.cc:671] Loaded metadata for table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7]
I20250831 01:55:30.868377 10863 catalog_manager.cc:671] Loaded metadata for table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]
I20250831 01:55:30.870638 10863 tablet_loader.cc:96] loaded metadata for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178])
I20250831 01:55:30.871210 10863 tablet_loader.cc:96] loaded metadata for tablet 0405e94d80a3470ab8944d584a85949c (table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5])
I20250831 01:55:30.873150 10863 tablet_loader.cc:96] loaded metadata for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7])
I20250831 01:55:30.873705 10863 tablet_loader.cc:96] loaded metadata for tablet 2e4ccddb53c642bc83e958032cc94c62 (table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d])
I20250831 01:55:30.875433 10863 tablet_loader.cc:96] loaded metadata for tablet 970335514c734553b2360365c81e2d51 (table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb])
I20250831 01:55:30.876204 10863 tablet_loader.cc:96] loaded metadata for tablet c2777d39a189458ea3e9326fe898c29b (table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa])
I20250831 01:55:30.876751 10863 tablet_loader.cc:96] loaded metadata for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792])
I20250831 01:55:30.877307 10863 tablet_loader.cc:96] loaded metadata for tablet cd278fc485cb44798d88871b367994fc (table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194])
I20250831 01:55:30.877867 10863 tablet_loader.cc:96] loaded metadata for tablet d3303431c6684d609b9238d908ad3cf4 (table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b])
I20250831 01:55:30.879730 10863 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 01:55:30.881491 10191 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:30.884250 10793 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:57593; Asking this server to re-register.
I20250831 01:55:30.885190 10191 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:30.886173 10863 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:30.886332 10863 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 01:55:30.887320 10793 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:30.889449 10863 catalog_manager.cc:1506] Loading token signing keys...
I20250831 01:55:30.890857 10863 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 0b3d4b29236443d8a14d363ec2aab7c8: Loaded TSK: 0
I20250831 01:55:30.891497 10863 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 01:55:30.892509 10863 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250831 01:55:30.893517 10863 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 23
F20250831 01:55:30.931782 9891 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 27, previous 27; dumping first 5 out of 7 received events: NotificationEvent(eventId=24, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_71f3e7f55e21496389aaf63260dec6a0, message={"server":"","servicePrincipal":"","db":"default","table":"table_71f3e7f55e21496389aaf63260dec6a0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_71f3e7f55e21496389aaf63260dec6a0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_71f3e7f55e21496389aaf63260dec6a0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_71f3e7f55e21496389aaf63260dec6a0\",\"kudu.table_id\":\"5dc2dd483179499eafb15a91231b58ce\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_54383364d8c14085a4963804ac2d6ba0, message={"server":"","servicePrincipal":"","db":"default","table":"table_54383364d8c14085a4963804ac2d6ba0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_54383364d8c14085a4963804ac2d6ba0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_54383364d8c14085a4963804ac2d6ba0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_54383364d8c14085a4963804ac2d6ba0\",\"kudu.table_id\":\"6b6b5b6d016b43f0921a4b22fe2aadd5\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_e5d7d8712d2d4ec181665ef359124956, message={"server":"","servicePrincipal":"","db":"default","table":"table_e5d7d8712d2d4ec181665ef359124956","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e5d7d8712d2d4ec181665ef359124956\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_e5d7d8712d2d4ec181665ef359124956\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e5d7d8712d2d4ec181665ef359124956\",\"kudu.table_id\":\"2f6da2a74ea0463397b0da181cd0b2c1\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_2faa27c5430c42e7b69a131d36238fdd, message={"server":"","servicePrincipal":"","db":"default","table":"table_2faa27c5430c42e7b69a131d36238fdd","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2faa27c5430c42e7b69a131d36238fdd\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_2faa27c5430c42e7b69a131d36238fdd\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2faa27c5430c42e7b69a131d36238fdd\",\"kudu.table_id\":\"9aa94d63ca7a4d22a2aefa09bbcf88cb\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_b0242810ea7a4fe3a1d3d8431eca51d2, message={"server":"","servicePrincipal":"","db":"default","table":"table_b0242810ea7a4fe3a1d3d8431eca51d2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b0242810ea7a4fe3a1d3d8431eca51d2\",\"kudu.table_id\":\"3e26e3f8edc04fa7a7ed3614967fb1aa\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fef85d15dcd google::LogMessage::Fail() at ??:0
@ 0x7fef85d19b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fef85d157cc google::LogMessage::Flush() at ??:0
@ 0x7fef85d15a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fef9c14a794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fef9c149041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250831 01:55:30.955174 9668 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_3bbd369a7fa546ba9996f1935a35e1cd" } new_table_name: "default.Table_d32a6a9a414a4a9593f6e87746a6193a" modify_external_catalogs: true
I20250831 01:55:30.955542 9677 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_11af32d34145411d9cddea34ecfb51de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:30.956386 9669 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_86b82dc71b1c42cdb1f2c620c2e2771a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:30.957788 9670 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_2faa27c5430c42e7b69a131d36238fdd" } modify_external_catalogs: true
I20250831 01:55:30.958042 9670 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38174:
table { table_name: "default.Table_2faa27c5430c42e7b69a131d36238fdd" } modify_external_catalogs: true
@ 0x7fef9c150cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250831 01:55:30.958233 9673 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_18f5062db5284a5d9fa9ecc471dc9af7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:30.958042 9674 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_a72e022123de434a95f24db4c575dd6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:30.959239 9672 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_b436ecb230034afea5e7f8d551228511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:30.959631 9676 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_71f3e7f55e21496389aaf63260dec6a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:30.960039 9671 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_e5d7d8712d2d4ec181665ef359124956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:30.960467 9675 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38174:
name: "default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fef9c150b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fef88a72cc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fef8be946db start_thread at ??:0
@ 0x7fef84bc671f clone at ??:0
I20250831 01:55:31.048888 10793 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:60889; Asking this server to re-register.
I20250831 01:55:31.048933 10146 raft_consensus.cc:1215] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250831 01:55:31.050101 10053 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:31.051918 10793 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
W20250831 01:55:31.141886 10079 connection.cc:537] client connection to 127.9.47.126:37509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:31.142671 9941 connection.cc:537] client connection to 127.9.47.126:37509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:31.142761 10334 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.47.126:37509)
W20250831 01:55:31.143151 10334 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.47.126:37509)
W20250831 01:55:31.143360 10334 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.47.126:37509)
W20250831 01:55:31.143764 10334 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.47.126:37509)
I20250831 01:55:31.174863 10146 raft_consensus.cc:1215] T 2e4ccddb53c642bc83e958032cc94c62 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250831 01:55:31.552821 10882 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:31.553054 10882 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:31.554817 10882 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:31.555833 10778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:31.557138 10780 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:31.557363 10778 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:31.557477 10778 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:31.557798 10882 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:32.059726 10893 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:32.059942 10893 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:32.060587 10893 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:32.062500 10780 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:32.062717 10778 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:32.062872 10778 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:32.063128 10893 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:32.627919 10904 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:32.628131 10904 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:32.628763 10904 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:32.630640 10778 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:32.630817 10780 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:32.630941 10780 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:32.631222 10904 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:32.785812 9405 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskz81FLs/build/asan/bin/kudu pid 10754
I20250831 01:55:32.798607 10824 generic_service.cc:196] Checking for leaks (request via RPC)
I20250831 01:55:32.905496 9405 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskz81FLs/build/asan/bin/kudu with pid 10754
I20250831 01:55:33.844976 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--webserver_interface=127.9.47.124
--webserver_port=42935
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250831 01:55:34.196506 10926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:34.196862 10926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:34.196995 10926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:34.209398 10926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:34.209553 10926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:34.209635 10926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:34.209703 10926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:34.223868 10926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.47.124
--webserver_port=42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:34.225479 10926 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:34.227507 10926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:34.235518 10932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:34.236099 10933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:34.236249 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:34.237057 10926 server_base.cc:1047] running on GCE node
I20250831 01:55:34.237717 10926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:34.239043 10926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:34.240247 10926 hybrid_clock.cc:648] HybridClock initialized: now 1756605334240198 us; error 36 us; skew 500 ppm
I20250831 01:55:34.240561 10926 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:34.242622 10926 webserver.cc:480] Webserver started at http://127.9.47.124:42935/ using document root <none> and password file <none>
I20250831 01:55:34.243301 10926 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:34.243403 10926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:34.248571 10926 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20250831 01:55:34.251904 10941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:34.253150 10926 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250831 01:55:34.253297 10926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:34.253999 10926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:34.290036 10926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:34.290897 10926 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:34.291177 10926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:34.310938 10926 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.124:37481
I20250831 01:55:34.310961 10992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.124:37481 every 8 connection(s)
I20250831 01:55:34.312580 10926 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
I20250831 01:55:34.313861 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 10926
I20250831 01:55:34.319875 10993 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:34.323623 10993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap starting.
I20250831 01:55:34.336784 10993 log.cc:826] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:34.385613 10993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=43 ignored=0} mutations{seen=47 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:34.386185 10993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap complete.
I20250831 01:55:34.391147 10993 raft_consensus.cc:357] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:34.391788 10993 raft_consensus.cc:738] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Initialized, Role: FOLLOWER
I20250831 01:55:34.392382 10993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 2.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:34.393124 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:34.393358 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:34.394059 10993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:34.406071 11012 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:34.406170 11012 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: loading cluster ID for follower catalog manager: success
I20250831 01:55:34.409190 11012 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: acquiring CA information for follower catalog manager: success
I20250831 01:55:34.412370 11012 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:34.409848 10926 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:34.415784 10926 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:34.485229 10191 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:34.488827 10958 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:54713; Asking this server to re-register.
I20250831 01:55:34.489581 10191 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:34.491561 10958 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:34.692652 10053 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:34.694312 10958 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:45259; Asking this server to re-register.
I20250831 01:55:34.694978 10053 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:34.696630 10958 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:34.722689 10325 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:34.724311 10958 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:46901; Asking this server to re-register.
I20250831 01:55:34.724998 10325 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:34.726805 10958 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:34.905411 11024 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:34.905706 11024 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:34.907534 11024 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:34.908550 10944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:34.910501 10944 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:34.911000 10942 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:34.911134 10942 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:34.911530 11024 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:35.421193 11031 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:35.421361 11031 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:35.422082 11031 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:35.424029 10942 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:35.424197 10944 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:35.424336 10944 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:35.424593 11031 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:36.139835 11040 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:36.140003 11040 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:36.140710 11040 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:36.142876 10944 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:36.143054 10942 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:36.143172 10942 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:36.143416 11040 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:36.685294 11052 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:36.685472 11052 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:36.686107 11052 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:36.687815 10942 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:36.687969 10944 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:36.688086 10944 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:36.688329 11052 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:36.733414 9405 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskz81FLs/build/asan/bin/kudu pid 10926
I20250831 01:55:36.740572 10988 generic_service.cc:196] Checking for leaks (request via RPC)
I20250831 01:55:36.842571 9405 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskz81FLs/build/asan/bin/kudu with pid 10926
W20250831 01:55:37.353677 10080 consensus_peers.cc:489] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 -> Peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Couldn't send request to peer 903e5d68fe5e4988a6b54623da741af5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3772a8213fd74d1a995e0c8eed42c08b. This is attempt 6: this message will repeat every 5th retry.
I20250831 01:55:37.599638 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--webserver_interface=127.9.47.124
--webserver_port=42935
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250831 01:55:37.920336 11074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:37.920670 11074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:37.920812 11074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:37.928965 11074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:37.929083 11074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:37.929170 11074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:37.929236 11074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:37.939327 11074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.47.124
--webserver_port=42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:37.941238 11074 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:37.943284 11074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:37.956591 11085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:37.956544 11082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:37.956622 11083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:37.957837 11074 server_base.cc:1047] running on GCE node
I20250831 01:55:37.958649 11074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:37.960148 11074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:37.961342 11074 hybrid_clock.cc:648] HybridClock initialized: now 1756605337961302 us; error 33 us; skew 500 ppm
I20250831 01:55:37.961687 11074 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:37.963889 11074 webserver.cc:480] Webserver started at http://127.9.47.124:42935/ using document root <none> and password file <none>
I20250831 01:55:37.964639 11074 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:37.964730 11074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:37.970108 11074 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20250831 01:55:37.972908 11091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:37.974309 11074 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250831 01:55:37.974484 11074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:37.975164 11074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:37.993794 11074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:37.994832 11074 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:37.995114 11074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:38.016310 11074 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.124:37481
I20250831 01:55:38.016310 11145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.124:37481 every 8 connection(s)
I20250831 01:55:38.018025 11074 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
I20250831 01:55:38.018440 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 11074
I20250831 01:55:38.026208 11146 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:38.030431 11146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap starting.
I20250831 01:55:38.044696 11146 log.cc:826] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:38.092370 11146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=43 ignored=0} mutations{seen=47 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:38.092896 11146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap complete.
I20250831 01:55:38.098049 11146 raft_consensus.cc:357] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:38.098742 11146 raft_consensus.cc:738] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Initialized, Role: FOLLOWER
I20250831 01:55:38.099330 11146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 2.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:38.100049 11150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:38.100320 11150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:38.101022 11146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:38.113749 11074 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:38.119186 11074 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:38.126039 11164 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:38.126214 11164 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: loading cluster ID for follower catalog manager: success
I20250831 01:55:38.130986 11164 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: acquiring CA information for follower catalog manager: success
I20250831 01:55:38.134299 11164 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:38.216753 10053 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:38.219266 11108 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:38699; Asking this server to re-register.
I20250831 01:55:38.219906 10053 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:38.221580 11108 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:38.245342 10325 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:38.247232 11108 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:60087; Asking this server to re-register.
I20250831 01:55:38.248143 10325 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:38.249706 11108 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:38.409453 10191 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:38.411769 11108 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:35757; Asking this server to re-register.
I20250831 01:55:38.412469 10191 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:38.413929 11108 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:38.499351 11150 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:38.499601 11150 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:38.501361 11150 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:38.502179 11092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:38.503762 11094 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:38.503990 11092 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:38.504186 11092 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:38.504505 11150 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 01:55:38.710438 10081 consensus_peers.cc:489] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 -> Peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): Couldn't send request to peer 67e9e49e7a864057adb3e685607c9844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3772a8213fd74d1a995e0c8eed42c08b. This is attempt 6: this message will repeat every 5th retry.
I20250831 01:55:39.009136 11176 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:39.009310 11176 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:39.010046 11176 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:39.012079 11092 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:39.012233 11094 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:39.012367 11094 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:39.012629 11176 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:39.911561 11193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:39.911777 11193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:39.912418 11193 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:39.916079 11094 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:39.916307 11092 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:39.916426 11092 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:39.916710 11193 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 01:55:41.219817 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--webserver_interface=127.9.47.125
--webserver_port=46501
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250831 01:55:41.522558 11220 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:41.522760 11220 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:41.523458 11220 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:41.526772 11092 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:41.526959 11094 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:41.527083 11094 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:41.527348 11220 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 01:55:41.548307 11217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:41.548635 11217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:41.548779 11217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:41.557912 11217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:41.558012 11217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:41.558105 11217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:41.558175 11217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:41.568729 11217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.47.125
--webserver_port=46501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:41.570555 11217 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:41.572647 11217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:41.580199 11231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:41.580160 11230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:41.581494 11233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:41.582020 11217 server_base.cc:1047] running on GCE node
I20250831 01:55:41.582719 11217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:41.584069 11217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:41.585290 11217 hybrid_clock.cc:648] HybridClock initialized: now 1756605341585238 us; error 36 us; skew 500 ppm
I20250831 01:55:41.585641 11217 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:41.587991 11217 webserver.cc:480] Webserver started at http://127.9.47.125:46501/ using document root <none> and password file <none>
I20250831 01:55:41.588739 11217 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:41.588850 11217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:41.594223 11217 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20250831 01:55:41.597148 11239 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:41.598451 11217 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250831 01:55:41.598603 11217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:41.599270 11217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:41.627259 11217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:41.628170 11217 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:41.628450 11217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:41.646416 11217 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.125:46791
I20250831 01:55:41.646440 11290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.125:46791 every 8 connection(s)
I20250831 01:55:41.647990 11217 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
I20250831 01:55:41.648720 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 11217
I20250831 01:55:41.655164 11291 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:41.658514 11291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap starting.
I20250831 01:55:41.670660 11291 log.cc:826] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:41.721671 11291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=64 ignored=0} inserts{seen=45 ignored=0} mutations{seen=49 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250831 01:55:41.722710 11291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap complete.
I20250831 01:55:41.732724 11291 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:41.745416 11291 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3cdeb22a7546b988ab9938f8dc1027, State: Initialized, Role: FOLLOWER
I20250831 01:55:41.746229 10055 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:41.747191 10327 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:41.748593 11291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 3.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:41.750638 11303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:41.751065 11303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:41.752656 11291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:41.754171 10193 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:41.765925 11317 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:41.766022 11317 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: loading cluster ID for follower catalog manager: success
I20250831 01:55:41.770323 11317 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: acquiring CA information for follower catalog manager: success
I20250831 01:55:41.773590 11317 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:41.774951 11217 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:41.780825 11217 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:42.253454 11256 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:40541; Asking this server to re-register.
I20250831 01:55:42.253413 11255 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:47191; Asking this server to re-register.
I20250831 01:55:42.254299 10055 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:42.254402 10327 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:42.255928 11256 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:42.256525 11255 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:42.258176 10193 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:42.259797 11255 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:57033; Asking this server to re-register.
I20250831 01:55:42.260352 10193 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:42.261436 11255 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:42.328076 11324 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:42.328341 11324 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:42.330237 11324 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:42.331338 11240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:42.333295 11240 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:42.337597 11118 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" candidate_term: 4 candidate_status { last_received { term: 3 index: 66 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905" is_pre_election: true
I20250831 01:55:42.338083 11118 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b3cdeb22a7546b988ab9938f8dc1027 in current term 4: Already voted for candidate 0b3d4b29236443d8a14d363ec2aab7c8 in this term.
I20250831 01:55:42.338826 11241 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:42.339188 11324 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 3 FOLLOWER]: Advancing to term 4
I20250831 01:55:42.342432 11324 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 01:55:43.034183 11334 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:43.034363 11334 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:43.035086 11334 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:43.035619 11118 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" candidate_term: 5 candidate_status { last_received { term: 3 index: 66 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905" is_pre_election: true
I20250831 01:55:43.035847 11118 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b3cdeb22a7546b988ab9938f8dc1027 in term 4.
I20250831 01:55:43.036262 11241 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027; no voters:
I20250831 01:55:43.036553 11334 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 01:55:43.036685 11334 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:43.036799 11334 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 4 FOLLOWER]: Advancing to term 5
W20250831 01:55:43.037118 11240 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:43.039144 11334 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:43.039629 11334 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 5 election: Requested vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
I20250831 01:55:43.040227 11118 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" candidate_term: 5 candidate_status { last_received { term: 3 index: 66 } } ignore_live_leader: false dest_uuid: "68251650a6ad4c1eba8826b2949c8905"
I20250831 01:55:43.040436 11118 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 4 FOLLOWER]: Advancing to term 5
W20250831 01:55:43.040783 11240 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:43.044167 11118 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b3cdeb22a7546b988ab9938f8dc1027 in term 5.
I20250831 01:55:43.044569 11241 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8
I20250831 01:55:43.044875 11334 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 01:55:43.045233 11334 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 5 LEADER]: Becoming Leader. State: Replica: 6b3cdeb22a7546b988ab9938f8dc1027, State: Running, Role: LEADER
I20250831 01:55:43.045646 11334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 3.66, Last appended by leader: 66, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:43.048128 11337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3cdeb22a7546b988ab9938f8dc1027. Latest consensus state: current_term: 5 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:43.048357 11337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:43.120791 11074 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:43.120985 11074 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:43.444119 11118 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Refusing update from remote peer 6b3cdeb22a7546b988ab9938f8dc1027: Log matching property violated. Preceding OpId in replica: term: 2 index: 61. Preceding OpId from leader: term: 5 index: 67. (index mismatch)
I20250831 01:55:43.444926 11337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 61, Time since last communication: 0.000s
I20250831 01:55:43.453799 11348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:43.454217 11348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:43.454217 11345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3cdeb22a7546b988ab9938f8dc1027. Latest consensus state: current_term: 5 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:43.454806 11345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:43.456815 11334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:43.457104 11334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:43.457351 11345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:43.457537 11345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
W20250831 01:55:43.458287 11240 consensus_peers.cc:489] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 -> Peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Couldn't send request to peer 0b3d4b29236443d8a14d363ec2aab7c8. Status: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:43.462200 11339 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 01:55:43.465059 11339 catalog_manager.cc:671] Loaded metadata for table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]
I20250831 01:55:43.465497 11339 catalog_manager.cc:671] Loaded metadata for table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]
I20250831 01:55:43.465864 11339 catalog_manager.cc:671] Loaded metadata for table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d]
I20250831 01:55:43.466249 11339 catalog_manager.cc:671] Loaded metadata for table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]
I20250831 01:55:43.466567 11339 catalog_manager.cc:671] Loaded metadata for table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]
I20250831 01:55:43.466921 11339 catalog_manager.cc:671] Loaded metadata for table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]
I20250831 01:55:43.467227 11339 catalog_manager.cc:671] Loaded metadata for table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]
I20250831 01:55:43.467685 11339 catalog_manager.cc:671] Loaded metadata for table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7]
I20250831 01:55:43.468007 11339 catalog_manager.cc:671] Loaded metadata for table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]
I20250831 01:55:43.469925 11339 tablet_loader.cc:96] loaded metadata for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178])
I20250831 01:55:43.470233 11339 tablet_loader.cc:96] loaded metadata for tablet 0405e94d80a3470ab8944d584a85949c (table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5])
I20250831 01:55:43.471225 11339 tablet_loader.cc:96] loaded metadata for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7])
I20250831 01:55:43.471501 11339 tablet_loader.cc:96] loaded metadata for tablet 2e4ccddb53c642bc83e958032cc94c62 (table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d])
I20250831 01:55:43.472316 11339 tablet_loader.cc:96] loaded metadata for tablet 970335514c734553b2360365c81e2d51 (table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb])
I20250831 01:55:43.472692 11339 tablet_loader.cc:96] loaded metadata for tablet c2777d39a189458ea3e9326fe898c29b (table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa])
I20250831 01:55:43.472977 11339 tablet_loader.cc:96] loaded metadata for tablet c2b779e787cc4e808f7b74558e733471 (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792])
I20250831 01:55:43.473239 11339 tablet_loader.cc:96] loaded metadata for tablet cd278fc485cb44798d88871b367994fc (table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194])
I20250831 01:55:43.473495 11339 tablet_loader.cc:96] loaded metadata for tablet d3303431c6684d609b9238d908ad3cf4 (table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b])
I20250831 01:55:43.474264 11339 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 01:55:43.475447 11339 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:43.475521 11339 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 01:55:43.476926 11339 catalog_manager.cc:1506] Loading token signing keys...
I20250831 01:55:43.478029 11339 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Loaded TSK: 0
I20250831 01:55:43.478883 11339 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 01:55:43.479952 11339 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250831 01:55:43.481103 11339 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 23
I20250831 01:55:43.556423 11255 master_service.cc:948] ReplaceTablet: received request to replace tablet 0fd06ceda8134e459d5bb86fac5cdbb8 from {username='slave'} at 127.0.0.1:49578
I20250831 01:55:43.556517 11256 master_service.cc:948] ReplaceTablet: received request to replace tablet c2b779e787cc4e808f7b74558e733471 from {username='slave'} at 127.0.0.1:49578
I20250831 01:55:43.557395 11254 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49578:
table { table_name: "default.Table_54383364d8c14085a4963804ac2d6ba0" } modify_external_catalogs: true
I20250831 01:55:43.557709 11254 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49578:
table { table_name: "default.Table_54383364d8c14085a4963804ac2d6ba0" } modify_external_catalogs: true
I20250831 01:55:43.558169 11249 master_service.cc:948] ReplaceTablet: received request to replace tablet c2b779e787cc4e808f7b74558e733471 from {username='slave'} at 127.0.0.1:49578
I20250831 01:55:43.558461 11250 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49578:
table { table_name: "default.Table_a3dfa51fdb0a42d390d8b0c78c181087" } new_table_name: "default.Table_f6625a8fb0a64ad5985877617c5a9e18" modify_external_catalogs: true
I20250831 01:55:43.558421 11251 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49578:
name: "default.Table_76700aef3a6e4f2abae81047ab54898d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:43.559003 11248 master_service.cc:948] ReplaceTablet: received request to replace tablet d36d4612593e4d8985b160918d54f87a from {username='slave'} at 127.0.0.1:49578
I20250831 01:55:43.559357 11253 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49578:
name: "default.Table_94d522ff92ac4ee7ae3906c381bfc49a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:43.560199 11247 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49578:
table { table_name: "default.Table_a91e9d30219f48539efe3ebbbf819119" } modify_external_catalogs: true
I20250831 01:55:43.560443 11247 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49578:
table { table_name: "default.Table_a91e9d30219f48539efe3ebbbf819119" } modify_external_catalogs: true
I20250831 01:55:43.560482 11248 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ee86091bc604b6a9d33c56d091fbd4d from {username='slave'} at 127.0.0.1:49578
I20250831 01:55:43.561363 11255 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49578:
name: "default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:43.557351 11252 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49578:
table { table_name: "default.Table_519e74debc8b417c8fe427b8b4c3bd29" } modify_external_catalogs: true
I20250831 01:55:43.561923 11252 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49578:
table { table_name: "default.Table_519e74debc8b417c8fe427b8b4c3bd29" } modify_external_catalogs: true
I20250831 01:55:43.562234 11248 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49578:
name: "default.Table_e5d7d8712d2d4ec181665ef359124956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250831 01:55:43.565311 11316 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 27, previous 27; dumping first 5 out of 7 received events: NotificationEvent(eventId=24, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_71f3e7f55e21496389aaf63260dec6a0, message={"server":"","servicePrincipal":"","db":"default","table":"table_71f3e7f55e21496389aaf63260dec6a0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_71f3e7f55e21496389aaf63260dec6a0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_71f3e7f55e21496389aaf63260dec6a0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_71f3e7f55e21496389aaf63260dec6a0\",\"kudu.table_id\":\"5dc2dd483179499eafb15a91231b58ce\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_54383364d8c14085a4963804ac2d6ba0, message={"server":"","servicePrincipal":"","db":"default","table":"table_54383364d8c14085a4963804ac2d6ba0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_54383364d8c14085a4963804ac2d6ba0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_54383364d8c14085a4963804ac2d6ba0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_54383364d8c14085a4963804ac2d6ba0\",\"kudu.table_id\":\"6b6b5b6d016b43f0921a4b22fe2aadd5\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_e5d7d8712d2d4ec181665ef359124956, message={"server":"","servicePrincipal":"","db":"default","table":"table_e5d7d8712d2d4ec181665ef359124956","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e5d7d8712d2d4ec181665ef359124956\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_e5d7d8712d2d4ec181665ef359124956\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e5d7d8712d2d4ec181665ef359124956\",\"kudu.table_id\":\"2f6da2a74ea0463397b0da181cd0b2c1\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_2faa27c5430c42e7b69a131d36238fdd, message={"server":"","servicePrincipal":"","db":"default","table":"table_2faa27c5430c42e7b69a131d36238fdd","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2faa27c5430c42e7b69a131d36238fdd\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_2faa27c5430c42e7b69a131d36238fdd\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2faa27c5430c42e7b69a131d36238fdd\",\"kudu.table_id\":\"9aa94d63ca7a4d22a2aefa09bbcf88cb\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_b0242810ea7a4fe3a1d3d8431eca51d2, message={"server":"","servicePrincipal":"","db":"default","table":"table_b0242810ea7a4fe3a1d3d8431eca51d2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b0242810ea7a4fe3a1d3d8431eca51d2\",\"kudu.table_id\":\"3e26e3f8edc04fa7a7ed3614967fb1aa\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250831 01:55:43.566390 11256 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Sending DeleteTablet for 3 replicas of tablet c2b779e787cc4e808f7b74558e733471
I20250831 01:55:43.566923 11249 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49578:
name: "default.Table_71f3e7f55e21496389aaf63260dec6a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f8fbc477dcd google::LogMessage::Fail() at ??:0
@ 0x7f8fbc47bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8fbc4777cc google::LogMessage::Flush() at ??:0
@ 0x7f8fbc477a11 google::LogMessage::~LogMessage() at ??:0
I20250831 01:55:43.588187 11256 catalog_manager.cc:6652] ReplaceTablet: tablet c2b779e787cc4e808f7b74558e733471 deleted and replaced by tablet 4a2211e0b5a24fc786e8f21ca3e04dfd
I20250831 01:55:43.588997 11256 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49578:
name: "default.Table_b436ecb230034afea5e7f8d551228511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:43.598675 10259 tablet_service.cc:1515] Processing DeleteTablet for tablet c2b779e787cc4e808f7b74558e733471 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a2211e0b5a24fc786e8f21ca3e04dfd) from {username='slave'} at 127.0.0.1:34378
@ 0x7f8fd28ac794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250831 01:55:43.604629 10258 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2211e0b5a24fc786e8f21ca3e04dfd (DEFAULT_TABLE table=default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.604831 10257 tablet_service.cc:1468] Processing CreateTablet for tablet 970335514c734553b2360365c81e2d51 (DEFAULT_TABLE table=default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.605108 10258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2211e0b5a24fc786e8f21ca3e04dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.605335 10257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 970335514c734553b2360365c81e2d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.605584 10256 tablet_service.cc:1468] Processing CreateTablet for tablet c2777d39a189458ea3e9326fe898c29b (DEFAULT_TABLE table=default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.606308 10256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2777d39a189458ea3e9326fe898c29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.609030 10260 tablet_service.cc:1468] Processing CreateTablet for tablet 0405e94d80a3470ab8944d584a85949c (DEFAULT_TABLE table=default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.609371 10260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0405e94d80a3470ab8944d584a85949c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.618625 10255 tablet_service.cc:1468] Processing CreateTablet for tablet d3303431c6684d609b9238d908ad3cf4 (DEFAULT_TABLE table=default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.619216 10255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3303431c6684d609b9238d908ad3cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.624298 9987 tablet_service.cc:1515] Processing DeleteTablet for tablet c2b779e787cc4e808f7b74558e733471 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a2211e0b5a24fc786e8f21ca3e04dfd) from {username='slave'} at 127.0.0.1:45704
I20250831 01:55:43.624545 9988 tablet_service.cc:1468] Processing CreateTablet for tablet d3303431c6684d609b9238d908ad3cf4 (DEFAULT_TABLE table=default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.624902 10125 tablet_service.cc:1515] Processing DeleteTablet for tablet c2b779e787cc4e808f7b74558e733471 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a2211e0b5a24fc786e8f21ca3e04dfd) from {username='slave'} at 127.0.0.1:37968
I20250831 01:55:43.625039 9988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3303431c6684d609b9238d908ad3cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.624994 9984 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2211e0b5a24fc786e8f21ca3e04dfd (DEFAULT_TABLE table=default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.625141 10121 tablet_service.cc:1468] Processing CreateTablet for tablet d3303431c6684d609b9238d908ad3cf4 (DEFAULT_TABLE table=default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.625345 9984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2211e0b5a24fc786e8f21ca3e04dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.625418 10126 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2211e0b5a24fc786e8f21ca3e04dfd (DEFAULT_TABLE table=default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.625735 10121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3303431c6684d609b9238d908ad3cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.626042 10124 tablet_service.cc:1468] Processing CreateTablet for tablet 0405e94d80a3470ab8944d584a85949c (DEFAULT_TABLE table=default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.626384 10124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0405e94d80a3470ab8944d584a85949c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.625741 10122 tablet_service.cc:1468] Processing CreateTablet for tablet c2777d39a189458ea3e9326fe898c29b (DEFAULT_TABLE table=default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.628729 10122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2777d39a189458ea3e9326fe898c29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.629176 9986 tablet_service.cc:1468] Processing CreateTablet for tablet 0405e94d80a3470ab8944d584a85949c (DEFAULT_TABLE table=default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.629513 9986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0405e94d80a3470ab8944d584a85949c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.630100 10123 tablet_service.cc:1468] Processing CreateTablet for tablet 970335514c734553b2360365c81e2d51 (DEFAULT_TABLE table=default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.630457 10123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 970335514c734553b2360365c81e2d51. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8fd28ab041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250831 01:55:43.634390 9985 tablet_service.cc:1468] Processing CreateTablet for tablet 970335514c734553b2360365c81e2d51 (DEFAULT_TABLE table=default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.634853 9985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 970335514c734553b2360365c81e2d51. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8fd28b2cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250831 01:55:43.638196 10126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2211e0b5a24fc786e8f21ca3e04dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 01:55:43.638177 9983 tablet_service.cc:1468] Processing CreateTablet for tablet c2777d39a189458ea3e9326fe898c29b (DEFAULT_TABLE table=default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250831 01:55:43.638697 9983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2777d39a189458ea3e9326fe898c29b. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8fd28b2b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f8fbf1d4cc9 kudu::Thread::SuperviseThread() at ??:0
I20250831 01:55:43.661229 11371 tablet_bootstrap.cc:492] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:43.663388 11371 tablet_bootstrap.cc:654] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.665360 11368 tablet_replica.cc:331] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: stopping tablet replica
I20250831 01:55:43.665580 11368 raft_consensus.cc:2241] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Raft consensus shutting down.
I20250831 01:55:43.665997 11368 raft_consensus.cc:2270] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:43.666613 11368 ts_tablet_manager.cc:1905] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f8fc25f66db start_thread at ??:0
@ 0x7f8fbb32871f clone at ??:0
I20250831 01:55:43.669701 11362 tablet_replica.cc:331] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: stopping tablet replica
I20250831 01:55:43.669939 11362 raft_consensus.cc:2241] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:43.670217 11362 raft_consensus.cc:2270] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:43.671588 11367 tablet_bootstrap.cc:492] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:43.674494 11362 ts_tablet_manager.cc:1905] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:43.675436 11369 tablet_replica.cc:331] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:43.675686 11369 raft_consensus.cc:2241] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:43.675978 11369 raft_consensus.cc:2270] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:43.676671 11369 ts_tablet_manager.cc:1905] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:43.678337 11367 tablet_bootstrap.cc:654] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.679049 11371 tablet_bootstrap.cc:492] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:43.679240 11371 ts_tablet_manager.cc:1397] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250831 01:55:43.681097 11370 tablet_bootstrap.cc:492] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:43.682911 11367 tablet_bootstrap.cc:492] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:43.683059 11367 ts_tablet_manager.cc:1397] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20250831 01:55:43.683826 11367 raft_consensus.cc:357] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.684011 11367 raft_consensus.cc:383] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.684091 11367 raft_consensus.cc:738] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.684305 11367 consensus_queue.cc:260] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.684861 11367 ts_tablet_manager.cc:1428] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250831 01:55:43.686093 11369 ts_tablet_manager.cc:1918] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:43.686236 11369 log.cc:1199] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/c2b779e787cc4e808f7b74558e733471
I20250831 01:55:43.686690 11369 ts_tablet_manager.cc:1939] T c2b779e787cc4e808f7b74558e733471 P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:43.688911 11367 tablet_bootstrap.cc:492] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:43.690949 11367 tablet_bootstrap.cc:654] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.691375 11370 tablet_bootstrap.cc:654] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.680246 11371 raft_consensus.cc:357] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.692325 11371 raft_consensus.cc:383] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.692436 11371 raft_consensus.cc:738] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.692759 11371 consensus_queue.cc:260] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.694101 11371 ts_tablet_manager.cc:1428] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20250831 01:55:43.694387 11371 tablet_bootstrap.cc:492] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:43.694646 11367 tablet_bootstrap.cc:492] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:43.694795 11367 ts_tablet_manager.cc:1397] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250831 01:55:43.695547 11367 raft_consensus.cc:357] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.695734 11367 raft_consensus.cc:383] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.695829 11367 raft_consensus.cc:738] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.696069 11371 tablet_bootstrap.cc:654] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.696070 11367 consensus_queue.cc:260] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.696585 11367 ts_tablet_manager.cc:1428] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250831 01:55:43.698794 11368 ts_tablet_manager.cc:1918] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:43.698925 11368 log.cc:1199] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-0/wal/wals/c2b779e787cc4e808f7b74558e733471
I20250831 01:55:43.699393 11368 ts_tablet_manager.cc:1939] T c2b779e787cc4e808f7b74558e733471 P 67e9e49e7a864057adb3e685607c9844: Deleting consensus metadata
I20250831 01:55:43.699664 11367 tablet_bootstrap.cc:492] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:43.701417 11367 tablet_bootstrap.cc:654] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.713552 11371 tablet_bootstrap.cc:492] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:43.713802 11371 ts_tablet_manager.cc:1397] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20250831 01:55:43.714823 11371 raft_consensus.cc:357] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.715041 11371 raft_consensus.cc:383] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.715149 11371 raft_consensus.cc:738] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.715770 11371 consensus_queue.cc:260] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.719205 11370 tablet_bootstrap.cc:492] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:43.719362 11370 ts_tablet_manager.cc:1397] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250831 01:55:43.719650 11371 ts_tablet_manager.cc:1428] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250831 01:55:43.720245 11370 raft_consensus.cc:357] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.720440 11370 raft_consensus.cc:383] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.720553 11370 raft_consensus.cc:738] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.720651 11367 tablet_bootstrap.cc:492] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:43.720800 11367 ts_tablet_manager.cc:1397] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.007s
I20250831 01:55:43.720845 11370 consensus_queue.cc:260] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.721573 11367 raft_consensus.cc:357] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.721820 11367 raft_consensus.cc:383] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.721907 11367 raft_consensus.cc:738] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.722112 11367 consensus_queue.cc:260] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.722819 11367 ts_tablet_manager.cc:1428] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250831 01:55:43.723117 11367 tablet_bootstrap.cc:492] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:43.724818 11367 tablet_bootstrap.cc:654] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.725853 11344 raft_consensus.cc:491] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:43.725991 11344 raft_consensus.cc:513] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.726840 11344 leader_election.cc:290] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:43.727219 11371 tablet_bootstrap.cc:492] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:43.727466 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "970335514c734553b2360365c81e2d51" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:43.728991 11371 tablet_bootstrap.cc:654] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.729285 11362 ts_tablet_manager.cc:1918] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:43.729404 11362 log.cc:1199] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-2/wal/wals/c2b779e787cc4e808f7b74558e733471
W20250831 01:55:43.729653 10215 leader_election.cc:343] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:43.729918 11362 ts_tablet_manager.cc:1939] T c2b779e787cc4e808f7b74558e733471 P 903e5d68fe5e4988a6b54623da741af5: Deleting consensus metadata
I20250831 01:55:43.731215 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "970335514c734553b2360365c81e2d51" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
W20250831 01:55:43.732925 10213 leader_election.cc:343] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:43.733088 10213 leader_election.cc:304] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5; no voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:43.734896 11344 raft_consensus.cc:2747] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 01:55:43.735436 11370 ts_tablet_manager.cc:1428] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250831 01:55:43.735801 11370 tablet_bootstrap.cc:492] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:43.737712 11370 tablet_bootstrap.cc:654] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.742913 11367 tablet_bootstrap.cc:492] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:43.742951 11370 tablet_bootstrap.cc:492] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:43.742967 11371 tablet_bootstrap.cc:492] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:43.743083 11367 ts_tablet_manager.cc:1397] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20250831 01:55:43.743098 11371 ts_tablet_manager.cc:1397] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20250831 01:55:43.743108 11370 ts_tablet_manager.cc:1397] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250831 01:55:43.743758 11367 raft_consensus.cc:357] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.743904 11367 raft_consensus.cc:383] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.743819 11370 raft_consensus.cc:357] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.743950 11367 raft_consensus.cc:738] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.743855 11371 raft_consensus.cc:357] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.744025 11371 raft_consensus.cc:383] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.744019 11370 raft_consensus.cc:383] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.744102 11370 raft_consensus.cc:738] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.744105 11371 raft_consensus.cc:738] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.744159 11367 consensus_queue.cc:260] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.744329 11371 consensus_queue.cc:260] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.744345 11370 consensus_queue.cc:260] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.744814 11370 ts_tablet_manager.cc:1428] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250831 01:55:43.744879 11371 ts_tablet_manager.cc:1428] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250831 01:55:43.745139 11370 tablet_bootstrap.cc:492] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:43.746057 11367 ts_tablet_manager.cc:1428] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250831 01:55:43.746392 11367 tablet_bootstrap.cc:492] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5: Bootstrap starting.
I20250831 01:55:43.746871 11370 tablet_bootstrap.cc:654] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.747979 11367 tablet_bootstrap.cc:654] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.756042 11371 tablet_bootstrap.cc:492] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:43.757954 11371 tablet_bootstrap.cc:654] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.762938 11371 tablet_bootstrap.cc:492] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:43.763106 11371 ts_tablet_manager.cc:1397] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20250831 01:55:43.763374 11367 tablet_bootstrap.cc:492] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5: No bootstrap required, opened a new log
I20250831 01:55:43.763574 11367 ts_tablet_manager.cc:1397] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20250831 01:55:43.763753 11370 tablet_bootstrap.cc:492] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:43.763976 11370 ts_tablet_manager.cc:1397] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.012s
I20250831 01:55:43.763990 11371 raft_consensus.cc:357] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.764209 11371 raft_consensus.cc:383] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.764309 11371 raft_consensus.cc:738] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.764626 11371 consensus_queue.cc:260] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.765040 11370 raft_consensus.cc:357] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.765153 11371 ts_tablet_manager.cc:1428] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250831 01:55:43.765044 11367 raft_consensus.cc:357] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.765230 11367 raft_consensus.cc:383] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.765336 11370 raft_consensus.cc:383] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.765367 11367 raft_consensus.cc:738] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903e5d68fe5e4988a6b54623da741af5, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.765494 11370 raft_consensus.cc:738] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.765666 11367 consensus_queue.cc:260] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.765820 11370 consensus_queue.cc:260] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.768283 11367 ts_tablet_manager.cc:1428] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250831 01:55:43.768801 11371 tablet_bootstrap.cc:492] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844: Bootstrap starting.
I20250831 01:55:43.769466 11370 ts_tablet_manager.cc:1428] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250831 01:55:43.770610 11371 tablet_bootstrap.cc:654] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.773730 11370 tablet_bootstrap.cc:492] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:43.774170 11393 raft_consensus.cc:491] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:43.774367 11393 raft_consensus.cc:513] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.778290 11370 tablet_bootstrap.cc:654] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.779608 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2777d39a189458ea3e9326fe898c29b" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:43.779688 11393 leader_election.cc:290] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:43.779870 10279 raft_consensus.cc:2466] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:43.780791 9942 leader_election.cc:304] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters:
I20250831 01:55:43.781038 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2777d39a189458ea3e9326fe898c29b" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
W20250831 01:55:43.782470 9941 leader_election.cc:343] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:43.782968 11393 raft_consensus.cc:2802] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:43.783152 11393 raft_consensus.cc:491] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:43.783291 11393 raft_consensus.cc:3058] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:43.787710 11371 tablet_bootstrap.cc:492] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844: No bootstrap required, opened a new log
I20250831 01:55:43.787889 11371 ts_tablet_manager.cc:1397] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.001s
I20250831 01:55:43.788034 11393 raft_consensus.cc:513] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.788774 11371 raft_consensus.cc:357] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.788977 11371 raft_consensus.cc:383] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.789199 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2777d39a189458ea3e9326fe898c29b" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:43.789336 11370 tablet_bootstrap.cc:492] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:43.789417 10279 raft_consensus.cc:3058] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:43.789502 11370 ts_tablet_manager.cc:1397] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.004s
I20250831 01:55:43.789819 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2777d39a189458ea3e9326fe898c29b" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:43.790357 11370 raft_consensus.cc:357] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.790647 11370 raft_consensus.cc:383] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.790767 11370 raft_consensus.cc:738] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.791106 11370 consensus_queue.cc:260] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.791689 11370 ts_tablet_manager.cc:1428] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250831 01:55:43.792058 11370 tablet_bootstrap.cc:492] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81: Bootstrap starting.
I20250831 01:55:43.792330 10279 raft_consensus.cc:2466] T c2777d39a189458ea3e9326fe898c29b P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
W20250831 01:55:43.790467 9941 leader_election.cc:343] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Illegal state: must be running to vote when last-logged opid is not known
I20250831 01:55:43.793032 9942 leader_election.cc:304] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters: fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:43.793305 11371 raft_consensus.cc:738] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.793596 10877 raft_consensus.cc:2802] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:43.793569 11371 consensus_queue.cc:260] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.794514 11370 tablet_bootstrap.cc:654] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81: Neither blocks nor log segments found. Creating new log.
I20250831 01:55:43.791227 11393 leader_election.cc:290] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:43.796530 11371 ts_tablet_manager.cc:1428] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250831 01:55:43.797186 10877 raft_consensus.cc:695] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:43.797561 10877 consensus_queue.cc:237] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.798296 11370 tablet_bootstrap.cc:492] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81: No bootstrap required, opened a new log
I20250831 01:55:43.798434 11370 ts_tablet_manager.cc:1397] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250831 01:55:43.799275 11370 raft_consensus.cc:357] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.799489 11370 raft_consensus.cc:383] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 01:55:43.799604 11370 raft_consensus.cc:738] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Initialized, Role: FOLLOWER
I20250831 01:55:43.799835 11370 consensus_queue.cc:260] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.800349 11370 ts_tablet_manager.cc:1428] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250831 01:55:43.874809 11364 raft_consensus.cc:491] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:43.874991 11364 raft_consensus.cc:513] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.875890 11364 leader_election.cc:290] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:43.876497 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2211e0b5a24fc786e8f21ca3e04dfd" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:43.876528 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2211e0b5a24fc786e8f21ca3e04dfd" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:43.876725 10279 raft_consensus.cc:2466] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:43.876745 10006 raft_consensus.cc:2466] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:43.877154 10081 leader_election.cc:304] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:43.877453 11364 raft_consensus.cc:2802] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:43.877571 11364 raft_consensus.cc:491] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:43.877669 11364 raft_consensus.cc:3058] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:43.879527 11364 raft_consensus.cc:513] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.880200 11364 leader_election.cc:290] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:43.880741 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2211e0b5a24fc786e8f21ca3e04dfd" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:43.880753 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2211e0b5a24fc786e8f21ca3e04dfd" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:43.880954 10279 raft_consensus.cc:3058] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:43.880972 10006 raft_consensus.cc:3058] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:43.883745 10006 raft_consensus.cc:2466] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:43.883798 10279 raft_consensus.cc:2466] T 4a2211e0b5a24fc786e8f21ca3e04dfd P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:43.884192 10080 leader_election.cc:304] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:43.884498 11329 raft_consensus.cc:2802] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:43.885679 11329 raft_consensus.cc:695] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:43.886050 11329 consensus_queue.cc:237] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
W20250831 01:55:43.906829 9941 connection.cc:537] server connection from 127.0.0.1:45704 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:43.906869 10082 connection.cc:537] server connection from 127.0.0.1:37968 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:43.906853 10215 connection.cc:537] server connection from 127.0.0.1:34378 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:43.907294 9943 connection.cc:537] client connection to 127.9.47.125:46791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:43.907771 10336 connection.cc:537] client connection to 127.9.47.125:46791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:43.908815 10336 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.47.125:46791)
W20250831 01:55:43.909483 10336 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.47.125:46791)
W20250831 01:55:43.909703 10336 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.47.125:46791)
W20250831 01:55:43.910090 10336 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.47.125:46791)
W20250831 01:55:43.910318 10215 connection.cc:537] client connection to 127.9.47.125:46791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250831 01:55:43.994457 11329 raft_consensus.cc:491] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:43.994661 11329 raft_consensus.cc:513] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.995257 11329 leader_election.cc:290] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:43.995757 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0405e94d80a3470ab8944d584a85949c" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:43.995801 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0405e94d80a3470ab8944d584a85949c" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
I20250831 01:55:43.995975 10279 raft_consensus.cc:2466] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:43.995986 10006 raft_consensus.cc:2466] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 0.
I20250831 01:55:43.996387 10080 leader_election.cc:304] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:43.996696 11329 raft_consensus.cc:2802] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:43.996817 11329 raft_consensus.cc:491] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:43.996949 11329 raft_consensus.cc:3058] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:43.998888 11329 raft_consensus.cc:513] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:43.999576 11329 leader_election.cc:290] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:43.999969 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0405e94d80a3470ab8944d584a85949c" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:44.000187 10279 raft_consensus.cc:3058] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.000349 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0405e94d80a3470ab8944d584a85949c" candidate_uuid: "fbe14365c42e4c6ba1173d829f997f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844"
I20250831 01:55:44.000541 10006 raft_consensus.cc:3058] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.002641 10006 raft_consensus.cc:2466] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:44.002674 10279 raft_consensus.cc:2466] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe14365c42e4c6ba1173d829f997f81 in term 1.
I20250831 01:55:44.003069 10081 leader_election.cc:304] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:44.003319 11329 raft_consensus.cc:2802] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:44.003492 11329 raft_consensus.cc:695] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Becoming Leader. State: Replica: fbe14365c42e4c6ba1173d829f997f81, State: Running, Role: LEADER
I20250831 01:55:44.003794 11329 consensus_queue.cc:237] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:44.027237 10877 raft_consensus.cc:491] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:44.027452 10877 raft_consensus.cc:513] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:44.028065 10877 leader_election.cc:290] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:44.028653 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "970335514c734553b2360365c81e2d51" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:44.028707 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "970335514c734553b2360365c81e2d51" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:44.028903 10146 raft_consensus.cc:2466] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:44.028949 10279 raft_consensus.cc:2466] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:44.029342 9941 leader_election.cc:304] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:44.029652 10877 raft_consensus.cc:2802] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:44.029735 10877 raft_consensus.cc:491] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:44.029891 10877 raft_consensus.cc:3058] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.031656 10877 raft_consensus.cc:513] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:44.032183 10877 leader_election.cc:290] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:44.032588 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "970335514c734553b2360365c81e2d51" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:44.032608 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "970335514c734553b2360365c81e2d51" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:44.032789 10279 raft_consensus.cc:3058] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.032800 10146 raft_consensus.cc:3058] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.035179 10279 raft_consensus.cc:2466] T 970335514c734553b2360365c81e2d51 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:44.035372 10146 raft_consensus.cc:2466] T 970335514c734553b2360365c81e2d51 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:44.035557 9942 leader_election.cc:304] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters:
I20250831 01:55:44.035837 10877 raft_consensus.cc:2802] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:44.036000 10877 raft_consensus.cc:695] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:44.036280 10877 consensus_queue.cc:237] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:44.119446 11406 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b3cdeb22a7546b988ab9938f8dc1027)
I20250831 01:55:44.119650 11406 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:44.120297 11406 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:44.121133 11092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250831 01:55:44.122196 11094 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:44.122351 11092 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:44.122463 11092 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:44.122704 11406 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 01:55:44.145438 10877 raft_consensus.cc:491] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:44.145670 10877 raft_consensus.cc:513] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:44.146502 10877 leader_election.cc:290] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:44.147097 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3303431c6684d609b9238d908ad3cf4" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81" is_pre_election: true
I20250831 01:55:44.147341 10146 raft_consensus.cc:2466] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:44.147785 9941 leader_election.cc:304] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:44.147749 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3303431c6684d609b9238d908ad3cf4" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:44.147991 10279 raft_consensus.cc:2466] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 0.
I20250831 01:55:44.148034 10877 raft_consensus.cc:2802] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 01:55:44.148135 10877 raft_consensus.cc:491] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:44.148227 10877 raft_consensus.cc:3058] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.150197 10877 raft_consensus.cc:513] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:44.150935 10877 leader_election.cc:290] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:44.151249 10279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3303431c6684d609b9238d908ad3cf4" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:44.151278 10146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3303431c6684d609b9238d908ad3cf4" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe14365c42e4c6ba1173d829f997f81"
I20250831 01:55:44.151461 10146 raft_consensus.cc:3058] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.151508 10279 raft_consensus.cc:3058] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.154345 10279 raft_consensus.cc:2466] T d3303431c6684d609b9238d908ad3cf4 P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:44.154342 10146 raft_consensus.cc:2466] T d3303431c6684d609b9238d908ad3cf4 P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
I20250831 01:55:44.154780 9941 leader_election.cc:304] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81; no voters:
I20250831 01:55:44.155122 10877 raft_consensus.cc:2802] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 01:55:44.156348 10877 raft_consensus.cc:695] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [term 1 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:44.156750 10877 consensus_queue.cc:237] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:44.174585 9405 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskz81FLs/build/asan/bin/kudu pid 11074
I20250831 01:55:44.185966 11141 generic_service.cc:196] Checking for leaks (request via RPC)
I20250831 01:55:44.319782 9405 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskz81FLs/build/asan/bin/kudu with pid 11074
I20250831 01:55:44.613494 10146 raft_consensus.cc:3058] T c2777d39a189458ea3e9326fe898c29b P fbe14365c42e4c6ba1173d829f997f81 [term 0 FOLLOWER]: Advancing to term 1
I20250831 01:55:44.616179 10877 consensus_queue.cc:1035] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:44.622170 11396 consensus_queue.cc:1035] T c2777d39a189458ea3e9326fe898c29b P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:44.710471 11329 consensus_queue.cc:1035] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:44.715924 11329 consensus_queue.cc:1035] T 4a2211e0b5a24fc786e8f21ca3e04dfd P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:44.929067 11427 consensus_queue.cc:1035] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:44.930461 11396 consensus_queue.cc:1035] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:44.935109 11427 consensus_queue.cc:1035] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:44.941712 11421 consensus_queue.cc:1035] T d3303431c6684d609b9238d908ad3cf4 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:45.000905 11421 consensus_queue.cc:1035] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:45.006614 11396 consensus_queue.cc:1035] T 970335514c734553b2360365c81e2d51 P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 01:55:45.207919 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--webserver_interface=127.9.47.124
--webserver_port=42935
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250831 01:55:45.550326 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:45.550626 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:45.550711 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:45.559141 11440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:45.559263 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:45.559355 11440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:45.559406 11440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:45.569109 11440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.47.124
--webserver_port=42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:45.571025 11440 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:45.572829 11440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:45.581112 11459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:45.581094 11458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:45.581274 11461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:45.582350 11440 server_base.cc:1047] running on GCE node
I20250831 01:55:45.583057 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:45.584394 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:45.585583 11440 hybrid_clock.cc:648] HybridClock initialized: now 1756605345585538 us; error 42 us; skew 500 ppm
I20250831 01:55:45.585969 11440 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:45.588002 11440 webserver.cc:480] Webserver started at http://127.9.47.124:42935/ using document root <none> and password file <none>
I20250831 01:55:45.588708 11440 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:45.588826 11440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:45.593933 11440 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250831 01:55:45.596591 11467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:45.597831 11440 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250831 01:55:45.597987 11440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:45.598672 11440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:45.613607 11440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:45.614421 11440 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:45.614670 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:45.634179 11440 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.124:37481
I20250831 01:55:45.634182 11518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.124:37481 every 8 connection(s)
I20250831 01:55:45.635809 11440 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
I20250831 01:55:45.642521 11519 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:45.644531 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 11440
I20250831 01:55:45.645931 11519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap starting.
I20250831 01:55:45.665465 11519 log.cc:826] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:45.695674 10191 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:45.720286 11519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=48 ignored=0} mutations{seen=57 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:45.721127 11519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap complete.
I20250831 01:55:45.726397 11519 raft_consensus.cc:357] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:45.727075 11519 raft_consensus.cc:738] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Initialized, Role: FOLLOWER
I20250831 01:55:45.727682 11519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 5.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:45.728423 11525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:45.728714 11525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:45.729391 11519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:45.747403 11539 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:45.747601 11539 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: loading cluster ID for follower catalog manager: success
I20250831 01:55:45.744052 11440 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:45.749791 11440 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:45.752530 11539 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: acquiring CA information for follower catalog manager: success
I20250831 01:55:45.755837 11539 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:45.976508 11484 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:58949; Asking this server to re-register.
I20250831 01:55:45.977281 10191 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:45.979013 11484 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:46.030639 10325 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:46.033032 11484 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:47481; Asking this server to re-register.
I20250831 01:55:46.033881 10325 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:46.035496 11484 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:46.047641 10053 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:46.051575 11484 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:35445; Asking this server to re-register.
I20250831 01:55:46.052727 10053 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:46.054821 11484 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:46.160460 11525 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:46.160751 11525 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:46.162628 11525 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:46.163244 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:46.164731 11470 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:46.164947 11468 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:46.165096 11468 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:46.165436 11525 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 01:55:46.713331 11556 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:46.713598 11556 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:46.714376 11556 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:46.716233 11468 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:46.716372 11470 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:46.716482 11468 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:46.716732 11556 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 01:55:47.333688 11566 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:47.333887 11566 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:47.334616 11566 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:47.336766 11470 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:47.336959 11468 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:47.337109 11468 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:47.337419 11566 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 01:55:47.836720 9405 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskz81FLs/build/asan/bin/kudu pid 11440
I20250831 01:55:47.843248 11514 generic_service.cc:196] Checking for leaks (request via RPC)
I20250831 01:55:47.948107 9405 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskz81FLs/build/asan/bin/kudu with pid 11440
W20250831 01:55:48.207727 10080 consensus_peers.cc:489] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 -> Peer 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173): Couldn't send request to peer 903e5d68fe5e4988a6b54623da741af5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3772a8213fd74d1a995e0c8eed42c08b. This is attempt 11: this message will repeat every 5th retry.
I20250831 01:55:48.564817 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--webserver_interface=127.9.47.124
--webserver_port=42935
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250831 01:55:48.739831 10081 consensus_peers.cc:489] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 -> Peer 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439): Couldn't send request to peer 67e9e49e7a864057adb3e685607c9844. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3772a8213fd74d1a995e0c8eed42c08b. This is attempt 11: this message will repeat every 5th retry.
W20250831 01:55:48.892226 11589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:48.892524 11589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:48.892611 11589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:48.903371 11589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:48.903517 11589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:48.903620 11589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:48.903699 11589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:48.914080 11589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.124:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.47.124
--webserver_port=42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:48.915666 11589 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:48.917410 11589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:48.926086 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:48.926395 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:48.926852 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:48.927331 11589 server_base.cc:1047] running on GCE node
I20250831 01:55:48.928095 11589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:48.929654 11589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:48.930846 11589 hybrid_clock.cc:648] HybridClock initialized: now 1756605348930790 us; error 50 us; skew 500 ppm
I20250831 01:55:48.931177 11589 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:48.933383 11589 webserver.cc:480] Webserver started at http://127.9.47.124:42935/ using document root <none> and password file <none>
I20250831 01:55:48.934173 11589 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:48.934283 11589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:48.939755 11589 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250831 01:55:48.942677 11609 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:48.944065 11589 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250831 01:55:48.944208 11589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
uuid: "68251650a6ad4c1eba8826b2949c8905"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:48.944870 11589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:48.966645 11589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:48.967518 11589 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:48.967849 11589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:48.986826 11660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.124:37481 every 8 connection(s)
I20250831 01:55:48.986826 11589 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.124:37481
I20250831 01:55:48.988714 11589 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-2/data/info.pb
I20250831 01:55:48.994067 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 11589
I20250831 01:55:48.995323 11661 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:48.999872 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap starting.
I20250831 01:55:49.016721 11661 log.cc:826] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:49.051379 10325 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:49.072273 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=48 ignored=0} mutations{seen=57 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:49.072854 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Bootstrap complete.
I20250831 01:55:49.078033 11661 raft_consensus.cc:357] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:49.078742 11661 raft_consensus.cc:738] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Initialized, Role: FOLLOWER
I20250831 01:55:49.079326 11661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 5.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:49.080090 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:49.080339 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:49.081035 11661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:49.090585 11589 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:49.096694 11589 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:49.097720 11679 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:49.097892 11679 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: loading cluster ID for follower catalog manager: success
I20250831 01:55:49.102424 11679 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: acquiring CA information for follower catalog manager: success
I20250831 01:55:49.104696 11679 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:49.151218 10053 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:49.154028 11626 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:38725; Asking this server to re-register.
I20250831 01:55:49.154765 10053 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:49.156543 11626 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:49.260100 10191 heartbeater.cc:344] Connected to a master server at 127.9.47.124:37481
I20250831 01:55:49.261823 11626 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:57269; Asking this server to re-register.
I20250831 01:55:49.262490 10191 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:49.263908 11626 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:49.555503 11626 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:42009; Asking this server to re-register.
I20250831 01:55:49.556155 10325 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:49.557650 11626 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:49.656876 11692 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:49.657131 11692 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:49.659034 11692 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:49.659901 11610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:49.661240 11612 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:49.661429 11610 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:49.661586 11610 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:49.661914 11692 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 01:55:50.308012 11701 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:50.308197 11701 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:50.308864 11701 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:50.310854 11612 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
W20250831 01:55:50.311017 11610 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:50.311156 11610 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:50.311394 11701 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 01:55:51.115008 11714 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:51.115168 11714 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:51.115808 11714 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:51.117704 11610 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:51.117921 11612 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791): Network error: Client connection negotiation failed: client connection to 127.9.47.125:46791: connect: Connection refused (error 111)
I20250831 01:55:51.118045 11612 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 6b3cdeb22a7546b988ab9938f8dc1027
I20250831 01:55:51.118273 11714 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 01:55:51.853094 9405 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskz81FLs/build/asan/bin/kudu
/tmp/dist-test-taskz81FLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--webserver_interface=127.9.47.125
--webserver_port=46501
--builtin_ntp_servers=127.9.47.84:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250831 01:55:52.181895 11725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250831 01:55:52.182204 11725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250831 01:55:52.182365 11725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250831 01:55:52.190770 11725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250831 01:55:52.190884 11725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250831 01:55:52.190953 11725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250831 01:55:52.191031 11725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250831 01:55:52.201500 11725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.47.84:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41285
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.47.125:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.47.125
--webserver_port=46501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 31 Aug 2025 01:43:22 UTC on 24a791456cd2
build id 7824
ASAN enabled
I20250831 01:55:52.203572 11725 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 01:55:52.205868 11725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 01:55:52.215148 11736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:52.215976 11737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 01:55:52.217844 11739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 01:55:52.218335 11725 server_base.cc:1047] running on GCE node
I20250831 01:55:52.219308 11725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250831 01:55:52.220814 11725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250831 01:55:52.222028 11725 hybrid_clock.cc:648] HybridClock initialized: now 1756605352221979 us; error 42 us; skew 500 ppm
I20250831 01:55:52.222363 11725 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 01:55:52.224427 11725 webserver.cc:480] Webserver started at http://127.9.47.125:46501/ using document root <none> and password file <none>
I20250831 01:55:52.225255 11725 fs_manager.cc:362] Metadata directory not provided
I20250831 01:55:52.225406 11725 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 01:55:52.231204 11725 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20250831 01:55:52.233932 11745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 01:55:52.235217 11725 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250831 01:55:52.235361 11725 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data,/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
format_stamp: "Formatted at 2025-08-31 01:55:21 on dist-test-slave-gcjs"
I20250831 01:55:52.236008 11725 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 01:55:52.272549 11725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 01:55:52.273521 11725 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 01:55:52.273890 11725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 01:55:52.296388 11725 rpc_server.cc:307] RPC server started. Bound to: 127.9.47.125:46791
I20250831 01:55:52.296406 11796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.47.125:46791 every 8 connection(s)
I20250831 01:55:52.298152 11725 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/master-1/data/info.pb
I20250831 01:55:52.304594 11797 sys_catalog.cc:263] Verifying existing consensus state
I20250831 01:55:52.304796 9405 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskz81FLs/build/asan/bin/kudu as pid 11725
I20250831 01:55:52.308862 11797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap starting.
I20250831 01:55:52.324308 11797 log.cc:826] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Log is configured to *not* fsync() on all Append() calls
I20250831 01:55:52.376448 11797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=48 ignored=0} mutations{seen=57 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 01:55:52.377013 11797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: Bootstrap complete.
I20250831 01:55:52.382640 11797 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:52.383301 11797 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3cdeb22a7546b988ab9938f8dc1027, State: Initialized, Role: FOLLOWER
I20250831 01:55:52.383900 11797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 5.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:52.384756 11801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:52.385002 11801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:52.385676 11797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 01:55:52.399811 11815 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:52.400015 11815 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: loading cluster ID for follower catalog manager: success
I20250831 01:55:52.394757 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:52.400790 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:52.405745 11815 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: acquiring CA information for follower catalog manager: success
I20250831 01:55:52.409323 11815 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250831 01:55:52.544553 10327 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:52.547647 11762 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" instance_seqno: 1756605324161103) as {username='slave'} at 127.9.47.67:43385; Asking this server to re-register.
I20250831 01:55:52.548432 10327 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:52.550274 11762 ts_manager.cc:194] Registered new tserver with Master: 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173)
I20250831 01:55:52.619441 10193 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:52.621568 11762 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" instance_seqno: 1756605323689029) as {username='slave'} at 127.9.47.66:58431; Asking this server to re-register.
I20250831 01:55:52.622210 10193 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:52.623548 11762 ts_manager.cc:194] Registered new tserver with Master: fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:52.662530 10055 heartbeater.cc:344] Connected to a master server at 127.9.47.125:46791
I20250831 01:55:52.664619 11762 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "67e9e49e7a864057adb3e685607c9844" instance_seqno: 1756605323188110) as {username='slave'} at 127.9.47.65:44205; Asking this server to re-register.
I20250831 01:55:52.665302 10055 heartbeater.cc:461] Registering TS with master...
I20250831 01:55:52.666720 11762 ts_manager.cc:194] Registered new tserver with Master: 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439)
I20250831 01:55:52.776077 11824 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 01:55:52.776335 11824 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:52.777016 11824 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
W20250831 01:55:52.779474 11610 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:52.783928 11772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 6 candidate_status { last_received { term: 5 index: 69 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" is_pre_election: true
I20250831 01:55:52.784515 11772 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68251650a6ad4c1eba8826b2949c8905 in term 5.
I20250831 01:55:52.785230 11612 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8
I20250831 01:55:52.785521 11824 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 01:55:52.785646 11824 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 01:55:52.785748 11824 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 5 FOLLOWER]: Advancing to term 6
I20250831 01:55:52.789659 11824 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:52.790263 11824 leader_election.cc:290] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 election: Requested vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 6b3cdeb22a7546b988ab9938f8dc1027 (127.9.47.125:46791)
I20250831 01:55:52.790793 11772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68251650a6ad4c1eba8826b2949c8905" candidate_term: 6 candidate_status { last_received { term: 5 index: 69 } } ignore_live_leader: false dest_uuid: "6b3cdeb22a7546b988ab9938f8dc1027"
I20250831 01:55:52.791028 11772 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 5 FOLLOWER]: Advancing to term 6
W20250831 01:55:52.791713 11610 leader_election.cc:336] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:52.794713 11772 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68251650a6ad4c1eba8826b2949c8905 in term 6.
I20250831 01:55:52.795117 11612 leader_election.cc:304] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68251650a6ad4c1eba8826b2949c8905, 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8
I20250831 01:55:52.795432 11824 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 01:55:52.795885 11824 raft_consensus.cc:695] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [term 6 LEADER]: Becoming Leader. State: Replica: 68251650a6ad4c1eba8826b2949c8905, State: Running, Role: LEADER
I20250831 01:55:52.796388 11824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 5.69, Last appended by leader: 69, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:52.799046 11828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68251650a6ad4c1eba8826b2949c8905. Latest consensus state: current_term: 6 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:52.799355 11828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [sys.catalog]: This master's current role is: LEADER
I20250831 01:55:52.800114 11830 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 01:55:52.802907 11830 catalog_manager.cc:671] Loaded metadata for table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa]
I20250831 01:55:52.803321 11830 catalog_manager.cc:671] Loaded metadata for table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5]
I20250831 01:55:52.803651 11830 catalog_manager.cc:671] Loaded metadata for table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d]
I20250831 01:55:52.803977 11830 catalog_manager.cc:671] Loaded metadata for table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178]
I20250831 01:55:52.804296 11830 catalog_manager.cc:671] Loaded metadata for table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b]
I20250831 01:55:52.804628 11830 catalog_manager.cc:671] Loaded metadata for table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb]
I20250831 01:55:52.804942 11830 catalog_manager.cc:671] Loaded metadata for table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792]
I20250831 01:55:52.805408 11830 catalog_manager.cc:671] Loaded metadata for table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7]
I20250831 01:55:52.805717 11830 catalog_manager.cc:671] Loaded metadata for table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]
I20250831 01:55:52.807539 11830 tablet_loader.cc:96] loaded metadata for tablet 0245596193a346d3aeb2d0aee88a6103 (table default.table_3bbd369a7fa546ba9996f1935a35e1cd [id=71f91154db86487a915ab787c1495178])
I20250831 01:55:52.807897 11830 tablet_loader.cc:96] loaded metadata for tablet 0405e94d80a3470ab8944d584a85949c (table default.table_54383364d8c14085a4963804ac2d6ba0 [id=6b6b5b6d016b43f0921a4b22fe2aadd5])
I20250831 01:55:52.808785 11830 tablet_loader.cc:96] loaded metadata for tablet 1fe4c7b8517747a9a9baef5c25d314f6 (table default.table_037da9465b8a41fda06bdeb1cf150d8b [id=b90d279f47a141ed846c461cd398d5e7])
I20250831 01:55:52.809082 11830 tablet_loader.cc:96] loaded metadata for tablet 2e4ccddb53c642bc83e958032cc94c62 (table default.table_a91e9d30219f48539efe3ebbbf819119 [id=6ca5ff2706e34ace8c9b111ac076142d])
I20250831 01:55:52.809520 11830 tablet_loader.cc:96] loaded metadata for tablet 4a2211e0b5a24fc786e8f21ca3e04dfd (table default.table_a3dfa51fdb0a42d390d8b0c78c181087 [id=ad1bd5ef25a343f0ad51d4e99ec44792])
I20250831 01:55:52.810281 11830 tablet_loader.cc:96] loaded metadata for tablet 970335514c734553b2360365c81e2d51 (table default.table_2faa27c5430c42e7b69a131d36238fdd [id=9aa94d63ca7a4d22a2aefa09bbcf88cb])
I20250831 01:55:52.810703 11830 tablet_loader.cc:96] loaded metadata for tablet c2777d39a189458ea3e9326fe898c29b (table default.table_b0242810ea7a4fe3a1d3d8431eca51d2 [id=3e26e3f8edc04fa7a7ed3614967fb1aa])
I20250831 01:55:52.811124 11830 tablet_loader.cc:96] loaded metadata for tablet cd278fc485cb44798d88871b367994fc (table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194])
I20250831 01:55:52.811388 11830 tablet_loader.cc:96] loaded metadata for tablet d3303431c6684d609b9238d908ad3cf4 (table default.table_513033900eee43b68b82843e69d0bb74 [id=9584cc0bbc7e437e967c3b8c39ab609b])
I20250831 01:55:52.812088 11830 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 01:55:52.813254 11830 catalog_manager.cc:1261] Loaded cluster ID: 8e25cdbc55124f669dfc6956e55f319e
I20250831 01:55:52.813338 11830 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 01:55:52.814774 11830 catalog_manager.cc:1506] Loading token signing keys...
I20250831 01:55:52.815824 11830 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Loaded TSK: 0
I20250831 01:55:52.816646 11830 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 01:55:52.817689 11830 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250831 01:55:52.818789 11830 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 23
I20250831 01:55:52.883052 10191 heartbeater.cc:499] Master 127.9.47.124:37481 was elected leader, sending a full tablet report...
I20250831 01:55:52.885102 11624 master_service.cc:948] ReplaceTablet: received request to replace tablet 0245596193a346d3aeb2d0aee88a6103 from {username='slave'} at 127.0.0.1:37824
I20250831 01:55:52.885530 11623 master_service.cc:948] ReplaceTablet: received request to replace tablet 0405e94d80a3470ab8944d584a85949c from {username='slave'} at 127.0.0.1:37824
I20250831 01:55:52.885614 11626 master_service.cc:948] ReplaceTablet: received request to replace tablet d36d4612593e4d8985b160918d54f87a from {username='slave'} at 127.0.0.1:37824
I20250831 01:55:52.886505 11622 master_service.cc:948] ReplaceTablet: received request to replace tablet 1def9b41fbe94b4ba680d0d4afd0d2b8 from {username='slave'} at 127.0.0.1:37824
I20250831 01:55:52.886783 11626 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37824:
name: "default.Table_76700aef3a6e4f2abae81047ab54898d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:52.887338 11622 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37824:
table { table_name: "default.Table_519e74debc8b417c8fe427b8b4c3bd29" } modify_external_catalogs: true
I20250831 01:55:52.887586 11622 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37824:
table { table_name: "default.Table_519e74debc8b417c8fe427b8b4c3bd29" } modify_external_catalogs: true
I20250831 01:55:52.888087 11620 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37824:
name: "default.Table_11af32d34145411d9cddea34ecfb51de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:52.888764 11618 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37824:
name: "default.Table_a72e022123de434a95f24db4c575dd6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:52.889242 11621 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37824:
table { table_name: "default.Table_3bbd369a7fa546ba9996f1935a35e1cd" } new_table_name: "default.Table_d32a6a9a414a4a9593f6e87746a6193a" modify_external_catalogs: true
I20250831 01:55:52.889534 11617 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37824:
table { table_name: "default.Table_2faa27c5430c42e7b69a131d36238fdd" } modify_external_catalogs: true
I20250831 01:55:52.889700 10121 tablet_service.cc:1515] Processing DeleteTablet for tablet 3772a8213fd74d1a995e0c8eed42c08b with delete_type TABLET_DATA_DELETED (replaced by tablet cd278fc485cb44798d88871b367994fc) from {username='slave'} at 127.0.0.1:37982
I20250831 01:55:52.889734 11617 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37824:
table { table_name: "default.Table_2faa27c5430c42e7b69a131d36238fdd" } modify_external_catalogs: true
I20250831 01:55:52.890358 11625 master_service.cc:948] ReplaceTablet: received request to replace tablet cd278fc485cb44798d88871b367994fc from {username='slave'} at 127.0.0.1:37824
I20250831 01:55:52.890201 11619 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37824:
name: "default.Table_86b82dc71b1c42cdb1f2c620c2e2771a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250831 01:55:52.892814 11836 tablet_replica.cc:331] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:52.893160 11836 raft_consensus.cc:2241] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Raft consensus shutting down.
F20250831 01:55:52.893338 11678 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 27, previous 27; dumping first 5 out of 7 received events: NotificationEvent(eventId=24, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_71f3e7f55e21496389aaf63260dec6a0, message={"server":"","servicePrincipal":"","db":"default","table":"table_71f3e7f55e21496389aaf63260dec6a0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_71f3e7f55e21496389aaf63260dec6a0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_71f3e7f55e21496389aaf63260dec6a0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_71f3e7f55e21496389aaf63260dec6a0\",\"kudu.table_id\":\"5dc2dd483179499eafb15a91231b58ce\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_54383364d8c14085a4963804ac2d6ba0, message={"server":"","servicePrincipal":"","db":"default","table":"table_54383364d8c14085a4963804ac2d6ba0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_54383364d8c14085a4963804ac2d6ba0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_54383364d8c14085a4963804ac2d6ba0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_54383364d8c14085a4963804ac2d6ba0\",\"kudu.table_id\":\"6b6b5b6d016b43f0921a4b22fe2aadd5\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_e5d7d8712d2d4ec181665ef359124956, message={"server":"","servicePrincipal":"","db":"default","table":"table_e5d7d8712d2d4ec181665ef359124956","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e5d7d8712d2d4ec181665ef359124956\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605327},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_e5d7d8712d2d4ec181665ef359124956\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e5d7d8712d2d4ec181665ef359124956\",\"kudu.table_id\":\"2f6da2a74ea0463397b0da181cd0b2c1\",\"transient_lastDdlTime\":\"1756605327\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_2faa27c5430c42e7b69a131d36238fdd, message={"server":"","servicePrincipal":"","db":"default","table":"table_2faa27c5430c42e7b69a131d36238fdd","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2faa27c5430c42e7b69a131d36238fdd\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_2faa27c5430c42e7b69a131d36238fdd\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2faa27c5430c42e7b69a131d36238fdd\",\"kudu.table_id\":\"9aa94d63ca7a4d22a2aefa09bbcf88cb\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=27, eventTime=1756605327, eventType=DROP_TABLE, dbName=default, tableName=table_b0242810ea7a4fe3a1d3d8431eca51d2, message={"server":"","servicePrincipal":"","db":"default","table":"table_b0242810ea7a4fe3a1d3d8431eca51d2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756605325},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/warehouse/table_b0242810ea7a4fe3a1d3d8431eca51d2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b0242810ea7a4fe3a1d3d8431eca51d2\",\"kudu.table_id\":\"3e26e3f8edc04fa7a7ed3614967fb1aa\",\"transient_lastDdlTime\":\"1756605325\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8e25cdbc55124f669dfc6956e55f319e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756605327}, messageFormat=json-0.2, catName=hive);
I20250831 01:55:52.893801 11836 raft_consensus.cc:2270] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
*** Check failure stack trace: ***
I20250831 01:55:52.894728 11836 ts_tablet_manager.cc:1905] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f54f54a2dcd google::LogMessage::Fail() at ??:0
@ 0x7f54f54a6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f54f54a27cc google::LogMessage::Flush() at ??:0
@ 0x7f54f54a2a11 google::LogMessage::~LogMessage() at ??:0
I20250831 01:55:52.903129 11836 ts_tablet_manager.cc:1918] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250831 01:55:52.903280 11836 log.cc:1199] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/3772a8213fd74d1a995e0c8eed42c08b
I20250831 01:55:52.903808 11836 ts_tablet_manager.cc:1939] T 3772a8213fd74d1a995e0c8eed42c08b P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:52.905635 11772 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Refusing update from remote peer 68251650a6ad4c1eba8826b2949c8905: Log matching property violated. Preceding OpId in replica: term: 5 index: 69. Preceding OpId from leader: term: 6 index: 71. (index mismatch)
W20250831 01:55:52.906315 11610 consensus_peers.cc:489] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 -> Peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Couldn't send request to peer 0b3d4b29236443d8a14d363ec2aab7c8. Status: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:52.906608 11610 catalog_manager.cc:4928] TS fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): tablet 3772a8213fd74d1a995e0c8eed42c08b (table default.table_03598ee84555460b9138a48936385e5e [id=b9f98af128a1402ca281bd1f8853c194]) successfully deleted
I20250831 01:55:52.906754 11824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
@ 0x7f550b8d7794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f550b8d6041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f550b8ddcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250831 01:55:52.919252 11840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68251650a6ad4c1eba8826b2949c8905. Latest consensus state: current_term: 6 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:52.919577 11840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f550b8ddb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250831 01:55:52.930428 11623 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Sending DeleteTablet for 3 replicas of tablet 0405e94d80a3470ab8944d584a85949c
I20250831 01:55:52.932169 11623 catalog_manager.cc:6652] ReplaceTablet: tablet 0405e94d80a3470ab8944d584a85949c deleted and replaced by tablet b8390e55901e45f0b7624c10c87ebe29
@ 0x7f54f81ffcc9 kudu::Thread::SuperviseThread() at ??:0
I20250831 01:55:52.934068 10121 tablet_service.cc:1515] Processing DeleteTablet for tablet 0405e94d80a3470ab8944d584a85949c with delete_type TABLET_DATA_DELETED (replaced by tablet b8390e55901e45f0b7624c10c87ebe29) from {username='slave'} at 127.0.0.1:37982
I20250831 01:55:52.934814 11836 tablet_replica.cc:331] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:52.935081 11836 raft_consensus.cc:2241] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 1 LEADER]: Raft consensus shutting down.
I20250831 01:55:52.935462 11836 raft_consensus.cc:2270] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:52.936218 11625 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 68251650a6ad4c1eba8826b2949c8905: Sending DeleteTablet for 3 replicas of tablet cd278fc485cb44798d88871b367994fc
I20250831 01:55:52.936242 11836 ts_tablet_manager.cc:1905] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f54fb6216db start_thread at ??:0
I20250831 01:55:52.936808 11625 catalog_manager.cc:6652] ReplaceTablet: tablet cd278fc485cb44798d88871b367994fc deleted and replaced by tablet dfc9dc914ade4bfcbb96a7a8643aa6d2
@ 0x7f54f435371f clone at ??:0
I20250831 01:55:52.938503 10121 tablet_service.cc:1515] Processing DeleteTablet for tablet cd278fc485cb44798d88871b367994fc with delete_type TABLET_DATA_DELETED (replaced by tablet dfc9dc914ade4bfcbb96a7a8643aa6d2) from {username='slave'} at 127.0.0.1:37982
I20250831 01:55:52.939538 11840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "68251650a6ad4c1eba8826b2949c8905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } } }
I20250831 01:55:52.939777 11840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [sys.catalog]: This master's current role is: FOLLOWER
I20250831 01:55:52.944036 11836 ts_tablet_manager.cc:1918] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:52.944198 11836 log.cc:1199] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/0405e94d80a3470ab8944d584a85949c
I20250831 01:55:52.944655 11836 ts_tablet_manager.cc:1939] T 0405e94d80a3470ab8944d584a85949c P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
I20250831 01:55:52.945914 11836 tablet_replica.cc:331] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: stopping tablet replica
I20250831 01:55:52.946138 11836 raft_consensus.cc:2241] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 01:55:52.946333 11836 raft_consensus.cc:2270] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 01:55:52.946986 11836 ts_tablet_manager.cc:1905] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250831 01:55:52.953889 11836 ts_tablet_manager.cc:1918] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250831 01:55:52.954104 11836 log.cc:1199] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Deleting WAL directory at /tmp/dist-test-taskz81FLs/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756605306376410-9405-0/minicluster-data/ts-1/wal/wals/cd278fc485cb44798d88871b367994fc
I20250831 01:55:52.954603 11836 ts_tablet_manager.cc:1939] T cd278fc485cb44798d88871b367994fc P fbe14365c42e4c6ba1173d829f997f81: Deleting consensus metadata
W20250831 01:55:53.117352 10080 connection.cc:537] server connection from 127.0.0.1:37982 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:53.117393 11749 connection.cc:537] server connection from 127.0.0.1:53888 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:53.117857 11853 negotiation.cc:337] Failed RPC negotiation. Trace:
0831 01:55:52.934446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.9.47.65:43439)
0831 01:55:52.937863 (+ 3417us) server_negotiation.cc:206] Beginning negotiation
0831 01:55:52.937872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0831 01:55:52.937897 (+ 25us) server_negotiation.cc:407] Connection header received
0831 01:55:52.937994 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0831 01:55:52.938001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0831 01:55:52.938560 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0831 01:55:52.938682 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0831 01:55:53.117463 (+178781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45716: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":3264,"thread_start_us":3329,"threads_started":1}
W20250831 01:55:53.118189 11851 negotiation.cc:337] Failed RPC negotiation. Trace:
0831 01:55:52.932685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.9.47.67:33173)
0831 01:55:52.942534 (+ 9849us) server_negotiation.cc:206] Beginning negotiation
0831 01:55:52.942542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0831 01:55:52.942565 (+ 23us) server_negotiation.cc:407] Connection header received
0831 01:55:52.942637 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0831 01:55:52.942643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0831 01:55:52.942702 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0831 01:55:52.942788 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0831 01:55:53.117911 (+175123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34388: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":9636,"thread_start_us":9502,"threads_started":1}
W20250831 01:55:53.118517 10214 connection.cc:537] client connection to 127.9.47.124:37481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:53.118541 10335 connection.cc:537] client connection to 127.9.47.124:37481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250831 01:55:53.119220 10335 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.47.124:37481)
W20250831 01:55:53.119598 10335 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.47.124:37481)
W20250831 01:55:53.120047 10335 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.47.124:37481)
W20250831 01:55:53.120448 10335 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.47.124:37481)
W20250831 01:55:53.241694 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:53.423532 11840 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:53.423753 11840 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:53.425891 11840 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:53.427114 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:53.429906 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:53.430218 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:53.430341 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:53.430619 11840 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:55:54.111402 11867 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:54.111588 11867 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:54.112229 11867 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:54.114126 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:54.598817 9405 master-stress-test.cc:577] Killing test thread 1/25
W20250831 01:55:55.118271 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:55.118433 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:55.119035 11877 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:55:55.197289 11874 raft_consensus.cc:491] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fbe14365c42e4c6ba1173d829f997f81)
I20250831 01:55:55.197484 11874 raft_consensus.cc:513] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:55.198173 11874 leader_election.cc:290] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67e9e49e7a864057adb3e685607c9844 (127.9.47.65:43439), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:55.198734 10006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0405e94d80a3470ab8944d584a85949c" candidate_uuid: "903e5d68fe5e4988a6b54623da741af5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67e9e49e7a864057adb3e685607c9844" is_pre_election: true
W20250831 01:55:55.199124 10213 leader_election.cc:343] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c
I20250831 01:55:55.199329 10215 leader_election.cc:304] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 903e5d68fe5e4988a6b54623da741af5; no voters: 67e9e49e7a864057adb3e685607c9844, fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:55.199574 11874 raft_consensus.cc:2747] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 01:55:55.680545 11881 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:55.680763 11881 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:55.681457 11881 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:55.683621 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:55.683787 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:55.683925 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:55.684206 11881 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:55:55.785526 11872 raft_consensus.cc:491] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fbe14365c42e4c6ba1173d829f997f81)
I20250831 01:55:55.785740 11872 raft_consensus.cc:513] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:55.786458 11872 leader_election.cc:290] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:55.786984 10276 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0405e94d80a3470ab8944d584a85949c" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5" is_pre_election: true
I20250831 01:55:55.787196 10276 raft_consensus.cc:2466] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 1.
W20250831 01:55:55.787269 9941 leader_election.cc:343] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c
I20250831 01:55:55.787586 9942 leader_election.cc:304] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters: fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:55.787859 11872 raft_consensus.cc:2802] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 01:55:55.787976 11872 raft_consensus.cc:491] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fbe14365c42e4c6ba1173d829f997f81)
I20250831 01:55:55.788077 11872 raft_consensus.cc:3058] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 1 FOLLOWER]: Advancing to term 2
I20250831 01:55:55.790174 11872 raft_consensus.cc:513] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
I20250831 01:55:55.790706 11872 leader_election.cc:290] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 2 election: Requested vote from peers 903e5d68fe5e4988a6b54623da741af5 (127.9.47.67:33173), fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085)
I20250831 01:55:55.791100 10276 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0405e94d80a3470ab8944d584a85949c" candidate_uuid: "67e9e49e7a864057adb3e685607c9844" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "903e5d68fe5e4988a6b54623da741af5"
I20250831 01:55:55.791313 10276 raft_consensus.cc:3058] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 1 FOLLOWER]: Advancing to term 2
W20250831 01:55:55.791424 9941 leader_election.cc:343] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c
I20250831 01:55:55.793094 10276 raft_consensus.cc:2466] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e9e49e7a864057adb3e685607c9844 in term 2.
I20250831 01:55:55.793435 9942 leader_election.cc:304] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67e9e49e7a864057adb3e685607c9844, 903e5d68fe5e4988a6b54623da741af5; no voters: fbe14365c42e4c6ba1173d829f997f81
I20250831 01:55:55.793689 11872 raft_consensus.cc:2802] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 01:55:55.793854 11872 raft_consensus.cc:695] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [term 2 LEADER]: Becoming Leader. State: Replica: 67e9e49e7a864057adb3e685607c9844, State: Running, Role: LEADER
I20250831 01:55:55.794142 11872 consensus_queue.cc:237] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 } } peers { permanent_uuid: "67e9e49e7a864057adb3e685607c9844" member_type: VOTER last_known_addr { host: "127.9.47.65" port: 43439 } } peers { permanent_uuid: "fbe14365c42e4c6ba1173d829f997f81" member_type: VOTER last_known_addr { host: "127.9.47.66" port: 46085 } }
W20250831 01:55:56.593043 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 1: this message will repeat every 5th retry.
I20250831 01:55:57.039032 10276 raft_consensus.cc:1273] T 0405e94d80a3470ab8944d584a85949c P 903e5d68fe5e4988a6b54623da741af5 [term 2 FOLLOWER]: Refusing update from remote peer 67e9e49e7a864057adb3e685607c9844: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 01:55:57.039508 11879 consensus_queue.cc:1035] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903e5d68fe5e4988a6b54623da741af5" member_type: VOTER last_known_addr { host: "127.9.47.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 01:55:57.189106 11898 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:57.189278 11898 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:57.189914 11898 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:57.191605 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:57.191807 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:57.191912 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:57.192173 11898 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:55:57.402140 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:55:57.402354 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:55:58.018363 11910 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:58.018623 11910 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:58.019572 11910 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:58.021790 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:55:58.021977 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:55:58.022140 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:58.022506 11910 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:55:59.356997 11929 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:55:59.357203 11929 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:55:59.357947 11929 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:55:59.358795 11747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250831 01:55:59.360023 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:55:59.360206 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:55:59.360324 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:55:59.360576 11929 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:56:02.404975 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:02.405158 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:56:03.469146 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 6: this message will repeat every 5th retry.
I20250831 01:56:04.206631 11986 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:04.206825 11986 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:04.207563 11986 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:04.209698 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:56:04.210107 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:56:04.210207 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:04.210464 11986 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:56:06.368710 12013 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:06.368934 12013 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:06.369585 12013 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:06.370671 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:56:06.371760 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:56:06.371937 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:56:06.372071 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:06.372326 12013 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:56:06.514273 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 6: this message will repeat every 5th retry.
I20250831 01:56:07.406738 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:07.406925 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:56:10.577672 12058 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:10.577883 12058 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:10.578442 12058 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:10.580372 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:56:10.580559 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:56:10.580673 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:10.580920 12058 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:56:12.408533 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:12.408727 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:56:13.127339 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 11: this message will repeat every 5th retry.
W20250831 01:56:16.344301 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 11: this message will repeat every 5th retry.
I20250831 01:56:17.410339 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:17.410528 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:56:20.416767 12158 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:20.416985 12158 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:20.417702 12158 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:20.418751 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:56:20.419983 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:56:20.420174 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:56:20.420344 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:20.420630 12158 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:56:22.251264 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 16: this message will repeat every 5th retry.
I20250831 01:56:22.411950 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:22.412160 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:56:24.896903 10054 heartbeater.cc:646] Failed to heartbeat to 127.9.47.126:37509 (160 consecutive failures): Network error: Failed to ping master at 127.9.47.126:37509: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:56:24.965297 10192 heartbeater.cc:646] Failed to heartbeat to 127.9.47.126:37509 (156 consecutive failures): Network error: Failed to ping master at 127.9.47.126:37509: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:56:25.137614 10325 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (65 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:56:26.405092 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 16: this message will repeat every 5th retry.
I20250831 01:56:27.413590 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:27.413836 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:56:32.363243 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 21: this message will repeat every 5th retry.
I20250831 01:56:32.415647 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:32.415840 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:56:37.175442 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 21: this message will repeat every 5th retry.
I20250831 01:56:37.417321 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:37.417488 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:56:39.696966 12359 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:39.697158 12359 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:39.698120 12359 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:39.699187 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:56:39.700282 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:56:39.700639 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:56:39.700783 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:39.701058 12359 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:56:42.011193 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 26: this message will repeat every 5th retry.
I20250831 01:56:42.419093 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:42.419277 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:56:43.264854 12391 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:43.265085 12391 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:43.265893 12391 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:43.267676 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:56:43.267884 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:56:43.268002 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:43.268208 12391 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:56:46.802516 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 26: this message will repeat every 5th retry.
I20250831 01:56:47.420717 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:47.420930 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:56:50.095592 12459 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:50.095803 12459 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:50.096508 12459 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:50.097453 11747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:56:50.098585 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:56:50.098788 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:56:50.098910 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:50.099149 12459 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:56:51.093362 12473 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:56:51.093551 12473 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:56:51.094209 12473 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:56:51.096328 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:56:51.096496 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:56:51.096625 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:56:51.096872 12473 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:56:51.641422 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 31: this message will repeat every 5th retry.
I20250831 01:56:52.422389 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:52.422601 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:56:56.448748 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 31: this message will repeat every 5th retry.
I20250831 01:56:57.423943 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:56:57.424130 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:57:01.263257 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 36: this message will repeat every 5th retry.
I20250831 01:57:02.425525 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:02.425705 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:57:06.097786 12628 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:57:06.097990 12628 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:57:06.098831 12628 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:57:06.099828 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:57:06.101294 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:57:06.101444 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:57:06.101609 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:57:06.101886 12628 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:57:06.151154 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 36: this message will repeat every 5th retry.
I20250831 01:57:07.427111 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:07.427297 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:57:08.950620 12649 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:57:08.950821 12649 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:57:08.951475 12649 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:57:08.953490 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:57:08.953673 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:57:08.953796 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:57:08.954022 12649 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:57:11.545888 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 41: this message will repeat every 5th retry.
I20250831 01:57:12.428834 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:12.429020 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:57:14.045418 12691 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:57:14.045634 12691 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:57:14.046358 12691 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:57:14.047256 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:57:14.048522 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:57:14.049240 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:57:14.049394 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:57:14.049777 12691 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:57:15.692298 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 41: this message will repeat every 5th retry.
I20250831 01:57:17.430212 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:17.430387 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:57:21.598733 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 46: this message will repeat every 5th retry.
I20250831 01:57:22.431754 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:22.431941 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:57:25.104342 10192 heartbeater.cc:646] Failed to heartbeat to 127.9.47.126:37509 (276 consecutive failures): Network error: Failed to ping master at 127.9.47.126:37509: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:57:25.137424 10053 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (273 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:57:25.334626 10325 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (185 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:57:26.111590 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 46: this message will repeat every 5th retry.
I20250831 01:57:27.433465 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:27.433686 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:57:31.082352 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 51: this message will repeat every 5th retry.
I20250831 01:57:31.957734 12873 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:57:31.957929 12873 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:57:31.958700 12873 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:57:31.959698 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:57:31.960899 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:57:31.961048 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:57:31.961186 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:57:31.961438 12873 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:57:32.434870 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:32.435060 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:57:36.624635 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 51: this message will repeat every 5th retry.
I20250831 01:57:37.436425 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:37.436651 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:57:40.757802 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 56: this message will repeat every 5th retry.
I20250831 01:57:42.438306 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:42.438473 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:57:46.132983 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 56: this message will repeat every 5th retry.
I20250831 01:57:47.440150 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:47.440361 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:57:49.833359 13057 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:57:49.833580 13057 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:57:49.834513 13057 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:57:49.835426 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:57:49.836537 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:57:49.836671 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:57:49.836846 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:57:49.837086 13057 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:57:50.646993 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 61: this message will repeat every 5th retry.
I20250831 01:57:52.440694 11751 maintenance_manager.cc:419] P 6b3cdeb22a7546b988ab9938f8dc1027: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033349
I20250831 01:57:52.441916 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:52.442101 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:57:52.505514 11750 maintenance_manager.cc:643] P 6b3cdeb22a7546b988ab9938f8dc1027: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.064s user 0.019s sys 0.005s Metrics: {"bytes_written":16883,"cfile_init":1,"dirs.queue_time_us":433,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":941,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":661,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":40,"thread_start_us":496,"threads_started":2,"wal-append.queue_time_us":323}
I20250831 01:57:52.506677 11751 maintenance_manager.cc:419] P 6b3cdeb22a7546b988ab9938f8dc1027: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3030 bytes on disk
I20250831 01:57:52.507396 11750 maintenance_manager.cc:643] P 6b3cdeb22a7546b988ab9938f8dc1027: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20250831 01:57:56.412632 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 61: this message will repeat every 5th retry.
I20250831 01:57:57.443377 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:57:57.443581 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:58:00.398321 13168 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:00.398536 13168 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:00.399268 13168 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:00.400267 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:58:00.401796 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:58:00.401968 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:58:00.402086 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:00.402312 13168 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:58:01.033282 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 66: this message will repeat every 5th retry.
I20250831 01:58:02.444931 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:02.445147 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:58:04.401182 13233 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:04.401417 13233 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:04.402151 13233 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:04.404560 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:58:04.404808 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:58:04.404932 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:04.405189 13233 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:58:05.667608 13257 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:05.667816 13257 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:05.668516 13257 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:05.669543 11747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:58:05.670754 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:58:05.670939 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:58:05.671039 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:05.671303 13257 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:58:07.100664 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 66: this message will repeat every 5th retry.
I20250831 01:58:07.446449 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:07.446635 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:58:12.310762 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 71: this message will repeat every 5th retry.
I20250831 01:58:12.448714 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:12.448942 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:58:17.450335 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:17.450523 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:58:18.224133 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 71: this message will repeat every 5th retry.
W20250831 01:58:22.368201 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 76: this message will repeat every 5th retry.
I20250831 01:58:22.452184 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:22.452373 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:58:25.241530 10192 heartbeater.cc:646] Failed to heartbeat to 127.9.47.126:37509 (396 consecutive failures): Network error: Failed to ping master at 127.9.47.126:37509: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:58:25.355214 10053 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (453 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:58:25.477258 13534 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:25.477485 13534 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:25.478397 13534 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:25.479511 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:58:25.480628 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:58:25.480774 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:58:25.480932 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:25.481212 13534 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:58:25.507390 10326 heartbeater.cc:646] Failed to heartbeat to 127.9.47.126:37509 (365 consecutive failures): Network error: Failed to ping master at 127.9.47.126:37509: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:58:27.453653 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:27.453856 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:58:28.329447 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 76: this message will repeat every 5th retry.
W20250831 01:58:32.141281 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 81: this message will repeat every 5th retry.
I20250831 01:58:32.455569 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:32.455739 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:58:37.395165 13698 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:37.395365 13698 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:37.396086 13698 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:37.397059 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:58:37.398212 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:58:37.398422 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:58:37.398557 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:37.398802 13698 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:58:37.456743 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:37.456928 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:58:38.024178 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 81: this message will repeat every 5th retry.
W20250831 01:58:42.036515 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 86: this message will repeat every 5th retry.
I20250831 01:58:42.458258 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:42.458437 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:58:43.359401 13784 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:43.359648 13784 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:43.360311 13784 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:43.361164 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:58:43.362543 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:58:43.362675 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:58:43.362812 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:43.363039 13784 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:58:44.134953 13798 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:44.135133 13798 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:44.135825 13798 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:44.137627 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:58:44.137804 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:58:44.137980 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:44.138255 13798 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:58:47.459913 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:47.460086 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:58:47.565732 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 86: this message will repeat every 5th retry.
W20250831 01:58:51.818010 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 91: this message will repeat every 5th retry.
I20250831 01:58:52.461364 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:52.461556 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:58:57.463001 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:58:57.463192 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:58:58.045118 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 91: this message will repeat every 5th retry.
I20250831 01:58:59.244277 14001 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:58:59.244479 14001 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:58:59.245321 14001 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:58:59.246363 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:58:59.247431 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:58:59.247593 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:58:59.247726 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:58:59.247972 14001 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:59:02.074632 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 96: this message will repeat every 5th retry.
I20250831 01:59:02.464777 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:02.465051 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:59:07.466580 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:07.466758 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:59:08.705649 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 96: this message will repeat every 5th retry.
I20250831 01:59:10.945132 14160 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:59:10.945348 14160 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:59:10.946103 14160 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:59:10.947052 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 01:59:10.948405 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:59:10.948567 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:59:10.948678 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:59:10.948942 14160 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:59:11.470520 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 101: this message will repeat every 5th retry.
I20250831 01:59:12.468412 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:12.468685 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:59:12.922797 14193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:59:12.922997 14193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:59:12.923708 14193 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:59:12.925388 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:59:12.925567 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:59:12.925700 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:59:12.925949 14193 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:59:17.470242 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:17.470484 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:59:17.745529 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 101: this message will repeat every 5th retry.
W20250831 01:59:21.790799 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 106: this message will repeat every 5th retry.
I20250831 01:59:22.472157 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:22.472374 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:59:25.377288 10192 heartbeater.cc:646] Failed to heartbeat to 127.9.47.126:37509 (516 consecutive failures): Network error: Failed to ping master at 127.9.47.126:37509: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:59:25.629317 10053 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (638 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:59:25.681589 10325 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (425 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:59:27.473824 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:27.474053 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:59:27.576762 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 106: this message will repeat every 5th retry.
I20250831 01:59:29.441500 14401 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:59:29.441890 14401 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:59:29.442867 14401 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:59:29.444025 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250831 01:59:29.445200 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 01:59:29.445375 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 01:59:29.445504 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:59:29.445750 14401 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:59:31.857334 14440 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:59:31.857517 14440 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:59:31.858208 14440 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:59:31.860174 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:59:31.860311 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:59:31.860455 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:59:31.860699 14440 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:59:32.002363 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 111: this message will repeat every 5th retry.
I20250831 01:59:32.475421 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:32.475611 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:59:33.266403 14463 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:59:33.266597 14463 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:59:33.267217 14463 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:59:33.269042 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:59:33.269202 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:59:33.269340 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:59:33.269563 14463 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 01:59:37.477248 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:37.477458 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:59:37.954291 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 111: this message will repeat every 5th retry.
W20250831 01:59:41.454185 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 116: this message will repeat every 5th retry.
I20250831 01:59:42.479301 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:42.479498 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:59:47.481098 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:47.481315 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:59:48.180111 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 116: this message will repeat every 5th retry.
I20250831 01:59:48.985407 14659 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 01:59:48.985605 14659 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 01:59:48.986575 14659 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 01:59:48.987408 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250831 01:59:48.988627 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 01:59:48.988801 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 01:59:48.988961 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 01:59:48.989198 14659 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 01:59:51.094461 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 121: this message will repeat every 5th retry.
I20250831 01:59:52.482915 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:52.483115 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 01:59:57.484550 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 01:59:57.484730 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 01:59:58.726176 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 121: this message will repeat every 5th retry.
I20250831 02:00:00.129849 14804 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 02:00:00.130054 14804 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 02:00:00.130795 14804 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 02:00:00.131718 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 02:00:00.132751 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 02:00:00.132901 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 02:00:00.133026 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 02:00:00.133277 14804 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 02:00:00.850553 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 126: this message will repeat every 5th retry.
I20250831 02:00:02.486238 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:02.486464 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 02:00:07.487937 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:07.488139 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 02:00:08.527856 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 126: this message will repeat every 5th retry.
W20250831 02:00:11.524991 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 131: this message will repeat every 5th retry.
I20250831 02:00:12.489480 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:12.489686 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 02:00:14.375948 14993 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 02:00:14.376178 14993 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 02:00:14.377017 14993 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 02:00:14.378033 11747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 02:00:14.379252 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 02:00:14.379451 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 02:00:14.379570 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 02:00:14.379830 14993 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 02:00:17.491081 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:17.491283 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 02:00:18.560228 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 131: this message will repeat every 5th retry.
I20250831 02:00:21.728489 15094 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 02:00:21.728811 15094 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
W20250831 02:00:21.731817 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20250831 02:00:21.732692 15094 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 02:00:21.733742 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 02:00:21.734164 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 02:00:21.734268 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 02:00:21.734520 15094 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 02:00:21.844584 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 136: this message will repeat every 5th retry.
I20250831 02:00:22.492820 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:22.493044 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 02:00:25.512111 10192 heartbeater.cc:646] Failed to heartbeat to 127.9.47.126:37509 (636 consecutive failures): Network error: Failed to ping master at 127.9.47.126:37509: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 02:00:25.862847 10053 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (827 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 02:00:25.869623 10325 heartbeater.cc:646] Failed to heartbeat to 127.9.47.124:37481 (545 consecutive failures): Network error: Failed to ping master at 127.9.47.124:37481: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 02:00:27.494645 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:27.494902 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 02:00:28.564193 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 136: this message will repeat every 5th retry.
W20250831 02:00:31.265344 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 141: this message will repeat every 5th retry.
I20250831 02:00:31.560992 15226 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 02:00:31.561215 15226 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 02:00:31.561977 15226 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 02:00:31.562928 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 02:00:31.564078 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 02:00:31.564273 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
I20250831 02:00:31.564391 11746 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 02:00:31.564608 15226 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 02:00:32.496310 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:32.496490 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 02:00:37.498349 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:37.498545 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 02:00:38.676564 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 141: this message will repeat every 5th retry.
W20250831 02:00:41.518366 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 146: this message will repeat every 5th retry.
I20250831 02:00:42.500272 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:42.500485 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 02:00:42.769132 15379 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 02:00:42.769335 15379 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
I20250831 02:00:42.770144 15379 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 02:00:42.770854 11747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250831 02:00:42.772042 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 02:00:42.772254 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 02:00:42.772396 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 02:00:42.772682 15379 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 02:00:47.502027 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:47.502277 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250831 02:00:48.462469 9941 consensus_peers.cc:489] T 0405e94d80a3470ab8944d584a85949c P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0405e94d80a3470ab8944d584a85949c. This is attempt 146: this message will repeat every 5th retry.
W20250831 02:00:51.238134 9941 consensus_peers.cc:489] T cd278fc485cb44798d88871b367994fc P 67e9e49e7a864057adb3e685607c9844 -> Peer fbe14365c42e4c6ba1173d829f997f81 (127.9.47.66:46085): Couldn't send request to peer fbe14365c42e4c6ba1173d829f997f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd278fc485cb44798d88871b367994fc. This is attempt 151: this message will repeat every 5th retry.
I20250831 02:00:52.504041 11725 master_runner.cc:186] Error getting master registration for 127.9.47.126:37509: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111),
I20250831 02:00:52.504283 11725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250831 02:00:52.514889 15515 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 68251650a6ad4c1eba8826b2949c8905)
I20250831 02:00:52.515115 15515 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b3d4b29236443d8a14d363ec2aab7c8" member_type: VOTER last_known_addr { host: "127.9.47.126" port: 37509 } } peers { permanent_uuid: "6b3cdeb22a7546b988ab9938f8dc1027" member_type: VOTER last_known_addr { host: "127.9.47.125" port: 46791 } } peers { permanent_uuid: "68251650a6ad4c1eba8826b2949c8905" member_type: VOTER last_known_addr { host: "127.9.47.124" port: 37481 } }
W20250831 02:00:52.517094 11746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20250831 02:00:52.518242 15515 leader_election.cc:290] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509), 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481)
W20250831 02:00:52.519595 11746 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0b3d4b29236443d8a14d363ec2aab7c8 (127.9.47.126:37509): Network error: Client connection negotiation failed: client connection to 127.9.47.126:37509: connect: Connection refused (error 111)
W20250831 02:00:52.519757 11747 leader_election.cc:336] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 68251650a6ad4c1eba8826b2949c8905 (127.9.47.124:37481): Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
I20250831 02:00:52.519876 11747 leader_election.cc:304] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b3cdeb22a7546b988ab9938f8dc1027; no voters: 0b3d4b29236443d8a14d363ec2aab7c8, 68251650a6ad4c1eba8826b2949c8905
I20250831 02:00:52.520125 15515 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6b3cdeb22a7546b988ab9938f8dc1027 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 02:00:53.022192 11751 maintenance_manager.cc:419] P 6b3cdeb22a7546b988ab9938f8dc1027: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3030 bytes on disk
I20250831 02:00:53.027081 11750 maintenance_manager.cc:643] P 6b3cdeb22a7546b988ab9938f8dc1027: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":284,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":87,"thread_start_us":476,"threads_started":2,"wal-append.queue_time_us":353}
W20250831 02:00:53.811929 10334 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 02:00:53.812186 10334 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 02:00:53.812384 10334 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
W20250831 02:00:53.812492 10334 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
F20250831 02:00:53.824316 10356 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Aborted at 1756605653 (unix time) try "date -d @1756605653" if you are using GNU date ***
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830494 10352 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830494 10352 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830595 10346 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94d522ff92ac4ee7ae3906c381bfc49a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830494 10352 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830595 10346 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94d522ff92ac4ee7ae3906c381bfc49a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831043 10360 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830494 10352 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830595 10346 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94d522ff92ac4ee7ae3906c381bfc49a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831043 10360 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831229 10343 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e5d7d8712d2d4ec181665ef359124956 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830494 10352 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830595 10346 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94d522ff92ac4ee7ae3906c381bfc49a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831043 10360 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831229 10343 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e5d7d8712d2d4ec181665ef359124956 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831293 10359 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e8000024bd) received by PID 9405 (TID 0x7f01098a6700) from PID 9405; stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830494 10352 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830595 10346 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94d522ff92ac4ee7ae3906c381bfc49a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831043 10360 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831229 10343 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e5d7d8712d2d4ec181665ef359124956 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831293 10359 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.832821 10344 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71f3e7f55e21496389aaf63260dec6a0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.832847 10367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20250831 02:00:53.824337 10362 master-stress-test.cc:391] Check failed: _s.ok() Bad status: F20250831 02:00:53.824322 Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_11af32d34145411d9cddea34ecfb51de on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825637 10350 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_86b82dc71b1c42cdb1f2c620c2e2771a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825723 10347 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a72e022123de434a95f24db4c575dd6b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.825942 10365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826112 10357 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.826164 10361 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828364 10342 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b436ecb230034afea5e7f8d551228511 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.828503 10345 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_18f5062db5284a5d9fa9ecc471dc9af7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829722 10348 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ece4b2c15acb4adb959d6ed6e42e2ab8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829804 10364 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.829989 10349 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_76700aef3a6e4f2abae81047ab54898d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830245 10363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830494 10352 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.830595 10346 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94d522ff92ac4ee7ae3906c381bfc49a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831043 10360 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831229 10343 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e5d7d8712d2d4ec181665ef359124956 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.831293 10359 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.832821 10344 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_71f3e7f55e21496389aaf63260dec6a0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)F20250831 02:00:53.832847 10367 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.47.126:37509,127.9.47.125:46791,127.9.47.124:37481, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.47.124:37481: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f0123262980 (unknown) at ??:0
@ 0x7f011c236fb7 gsignal at ??:0
@ 0x7f011c238921 abort at ??:0
@ 0x7f011ef6edcd google::LogMessage::Fail() at ??:0
@ 0x7f011ef72b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f011ef6e7cc google::LogMessage::Flush() at ??:0
@ 0x7f011ef6ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40c481 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:328
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f011c8be6df (unknown) at ??:0
@ 0x7f01232576db start_thread at ??:0
@ 0x7f011c31971f clone at ??:0